Robot setting apparatus and robot setting method

ABSTRACT

A robot setting apparatus includes a grip reference point setting unit, a grip direction setting unit that defines a grip direction in which the end effector model grips the workpiece model, a workpiece side grip location designation unit that designates a grip position at which the end effector model grips the workpiece model in a state in which at least the workpiece model is displayed in an image display region, and a relative position setting unit that sets a relative position between the end effector model and the workpiece model such that the grip direction defined in the grip direction setting unit is orthogonal to a workpiece plane representing an attitude of the workpiece model displayed in the image display region, and the grip reference point is located at the grip position along the grip direction.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims foreign priority based on Japanese PatentApplication No. 2017-040979, filed Mar. 3, 2017, the contents of whichis incorporated herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a robot setting apparatus, and a robotsetting method, and particularly to a robot setting apparatus and arobot setting method, for controlling a robot in which athree-dimensional shape of each of a plurality of workpieces which arepicking targets stacked in a work space is measured with a sensor unit,and a bin picking operation of sequentially gripping and taking out theworkpieces with an end effector provided at the tip of an arm portion ofthe robot is performed.

2. Description of Related Art

A robot device has been developed which has a manipulator combined witha robot vision, can image a target workpiece with the robot vision suchthat height information is acquired, can grip (pick) the workpiece at anappropriate position, and can place the workpiece at a desired position.By using such a robot device, an operation called bin picking isperformed in which a plurality of workpieces put in a returnable box areimaged with a camera or a sensor unit forming the robot vision such thatattitudes thereof are recognized and thus an appropriate grip positionis recognized, then the arm of the robot is moved to the grip position,and a workpiece is gripped by the end effector such as a hand portionprovided at the tip of the arm, and is placed at a determined positionoutside the returnable box.

In the bin picking using such a robot vision, a positional relationshipbetween a workpiece and the robot of when the workpiece is caught isregistered as a grip position, a grip position of the robot for aworkpiece detected with the robot vision is calculated, and the robot ismoved to the calculated position such that the workpiece is picked.

Such work of registering a grip position is required for a user toperform alignment with an attitude of gripping a workpiece with thenaked eyes while manually moving an end effector. However, the work isconsiderably troublesome since the number of parameters which can beadjusted is large, and thus a degree of freedom increases, depending onadjustment of a position or an attitude of the end effector.

Examples of the related art include Japanese Patents 3782679 and4962123.

SUMMARY OF THE INVENTION

The present invention has been made in light of the circumstances, andan object thereof is to provide a robot setting apparatus, a robotsetting method, a robot setting program, a computer readable recordingmedium, and an apparatus storing the program, capable of easilyperforming work of designating a grip state in an end effector when arobot device is set.

According to a first aspect of the present invention, there is provideda robot setting apparatus controlling a robot performing a bin pickingoperation of a sensor unit measuring a three-dimensional shape of eachof a plurality of workpieces stacked in a work space and sequentiallytaking out the workpieces with an end effector provided at a tip of anarm portion of the robot, the robot setting apparatus including aworkpiece model registration unit that registers a workpiece modelvirtually expressing a three-dimensional shape of a workpiece withthree-dimensional CAD data or a height image; an end effector modelregistration unit that registers an end effector model virtuallyexpressing a three-dimensional shape of an end effector withthree-dimensional CAD data; an image display region in which the endeffector model and the workpiece model are displayed on a virtualthree-dimensional space; a grip reference point setting unit thatdefines a grip reference point corresponding to a position at which theworkpiece model is gripped for the end effector model; a grip directionsetting unit that defines a grip direction in which the end effectormodel grips the workpiece model; a workpiece side grip locationdesignation unit that designates a grip position at which the endeffector model grips the workpiece model in a state in which at leastthe workpiece model is displayed in the image display region; and arelative position setting unit that sets a relative position between theend effector model and the workpiece model such that the grip directiondefined in the grip direction setting unit is orthogonal to a workpieceplane representing an attitude of the workpiece model displayed in theimage display region, and the grip reference point is located at thegrip position along the grip direction. With this configuration, in acase where a grip state in which a workpiece is gripped by an endeffector is simulated, it is possible to obtain an advantage thatsetting work for a position at which a workpiece model is gripped by anend effector model can be easily performed. Particularly, since a gripdirection is orthogonal to a workpiece plane, and a grip reference pointand a grip position are located on an axis in the grip direction, an endeffector model has only to come close to a workpiece model along thegrip direction such that the grip position can be adjusted, and thus itis possible to obtain an advantage that a work burden on a user side isconsiderably reduced.

According to the robot setting apparatus related to a second aspect, inaddition to the configuration, the relative position setting unitautomatically may adjusts the relative position between the end effectormodel and the workpiece model such that the grip direction is orthogonalto the workpiece plane, and the grip reference point and the gripposition are located on an axis along the grip direction. With thisconfiguration, a grip reference position and a grip direction aredefined on an end effector model side in advance, and the grip directionis set to be orthogonal to a workpiece plane, and a grip reference pointof the end effector model and a grip position of a workpiece are set tobe located on an axis in the grip direction. Thus, a movement directionof the end effector model is defined, and a user can obtain a grip stateby adjusting only a distance between the end effector model and aworkpiece model. As a result, considerable labor-saving for fitting workbetween grip positions of an end effector model and a workpiece model,which is troublesome work in the related art, can be expected.

According to the robot setting apparatus related to a third aspect, inaddition to any one of the configuration, the relative position settingunit may move the end effector model along the grip direction until theend effector model interferes with the workpiece model, andautomatically defines a grip state at an attitude of being separatedfrom a position reaching an interference position by a predetermineddistance, in a state in which the grip direction is orthogonal to theworkpiece plane, and the grip reference point and the grip position arelocated on an axis along the grip direction by adjusting the relativeposition between the end effector model and the workpiece model. Withthis configuration, it is possible to obtain an advantage that automaticadjustment to a position and an attitude at which a workpiece model isgripped by an end effector model can be performed, and thus a workburden on a user side can be further reduced.

According to the robot setting apparatus related to a fourth aspect, inaddition to any one of the configuration, the robot setting apparatusmay further include a search model registration unit that registers asearch model which is used to perform a three-dimensional search forspecifying an attitude and a position of each workpiece included in aninput image from the input image indicating a state in which a pluralityof workpiece groups are loaded in bulk, and which virtually expresses athree-dimensional shape of a workpiece; a three-dimensional search unitthat performs a three-dimensional search for specifying an attitude anda position of each workpiece from the input image by using the searchmodel registered by the search model registration unit; and athree-dimensional pick determination unit that determines whether or nota workpiece can be gripped by an end effector at a grip positiondesignated for the workpiece by the workpiece side grip locationdesignation unit on the basis of a search result in the input imagesearched by the three-dimensional search unit.

According to the robot setting apparatus related to a fifth aspect, inaddition to any one of the configuration, the robot setting apparatusmay further include an input image acquisition unit that acquires aninput image including a three-dimensional shape on the basis of an imageof a plurality of workpiece groups measured in the sensor unit, and thethree-dimensional search unit may perform a three-dimensional search forspecifying an attitude and a position of each workpiece from the inputimage acquired by the input image acquisition unit by using the searchmodel registered by the search model registration unit. With thisconfiguration, a three-dimensional search can be performed from an inputimage acquired by actually imaging a workpiece, and thus it is possibleto perform grip determination conforming more to the actualcircumstances.

According to the robot setting apparatus related to a sixth aspect, inaddition to any one of the configuration, the search model registrationunit and the workpiece model registration unit may be configured byusing a common member. With this configuration, a model regarding asingle workpiece is registered, and can thus be used in common toregistration of a grip position and registration of a search model for athree-dimensional search, and thus it is possible to obtain an advantagethat setting can be simplified.

According to the robot setting apparatus related to a seventh aspect, inaddition to any one of the configuration, the grip reference pointsetting unit may set the grip reference point to a preset predeterminedvalue and/or the grip direction setting unit sets the grip direction toa preset predetermined value.

According to the robot setting apparatus related to an eighth aspect, inaddition to any one of the configuration, the grip reference pointsetting unit may allow a user to set the grip reference point and/or thegrip direction setting unit allows a user to set the grip direction.

According to the robot setting apparatus related to a ninth aspect, inaddition to any one of the configuration, a grip reference point and agrip direction passing through the grip reference point may be displayedto overlap the end effector model in the image display region. With thisconfiguration, a movement direction for making an end effector modelcome close to a workpiece model can be presented to a user in a betterunderstanding manner, and thus it is possible to provide an environmentin which grip position adjustment work is facilitated to the user.

According to the robot setting apparatus related to a tenth aspect, inaddition to any one of the configuration, the robot setting apparatusmay further include a workpiece grip propriety display region in which adetermination result of grip propriety at a grip position designated foreach workpiece in the three-dimensional pick determination unit isdisplayed; and a workpiece grip impossibility cause display region inwhich a cause of grip impossibility for a grip position which isdetermined as grip being impossible at the grip position designated foreach workpiece in the three-dimensional pick determination unit isdisplayed. With this configuration, a cause of being incapable ofgripping a workpiece is displayed, and this contributes to resetting ofa grip position, for example, since a user easily examines which gripposition is preferably added.

According to the robot setting apparatus related to an eleventh aspect,in addition to any one of the configuration, the three-dimensional pickdetermination unit may include an interference determination unit thatdetermines the presence or absence of interference with a member presentaround a workpiece at a grip position designated for the workpiece bythe workpiece side grip location designation unit on the basis of asearch result of each workpiece searched for by the three-dimensionalsearch unit, and the three-dimensional pick determination unit maydetermine that the workpiece determined as there being interference bythe interference determination unit cannot be gripped.

According to the robot setting apparatus related to a twelfth aspect, inaddition to any one of the configuration, the robot setting apparatusmay further include an inclined angle setting unit that sets anallowable inclined angle range for an attitude of a workpiece, theinterference determination unit may include an angle determination unitthat determines whether or not an attitude of a search result of aworkpiece searched for by the three-dimensional search unit is includedin an inclined angle range set by the inclined angle setting unit, andthe three-dimensional pick determination unit may determine that theworkpiece cannot be gripped in a case where the angle determination unitdetermines that the attitude of the search result of the workpiecesearched for by the three-dimensional search unit is not included in theinclined angle range set by the inclined angle setting unit. With thisconfiguration, in a case where an attitude of a workpiece is too steep,and thus the accuracy of three-dimensional measurement cannot beexpected, this is excluded such that wrong selection or wrongdetermination of a grip position can be prevented, and thus it ispossible to increase reliability.

According to the robot setting apparatus related to a thirteenth aspect,in addition to any one of the configuration, a cause of gripimpossibility displayed in the workpiece grip impossibility causedisplay region may include at least one of an end effector modelinterfering with an object present around a workpiece and an inclinedangle of an end effector model exceeding a predetermined range. Withthis configuration, a cause such as interference or an attitude of anend effector model at a grip position at which grip impossibility isdetermined can be specifically specified, and thus a user easily takes ameasure according thereto.

According to the robot setting apparatus related to a fourteenth aspect,in addition to any one of the configuration, the workpiece side griplocation designation unit may register a plurality of grip positions fora workpiece model.

According to the robot setting apparatus related to a fifteenth aspect,in addition to any one of the configuration, the robot setting apparatusmay further include a grip solution candidate display region in whichgrip positions set for any one of search results of one or moreworkpieces searched for by the three-dimensional search unit aredisplayed in a list form.

According to the robot setting apparatus related to a sixteenth aspect,in addition to any one of the configuration, a position and an attitudeof an end effector model corresponding to a grip position selected inthe grip solution candidate display region may be displayed in the imagedisplay region. With this configuration, positions or attitudes of anend effector model are displayed in a switching manner for each gripposition, and thus a user can visually recognize a grip state.

According to the robot setting apparatus related to a seventeenthaspect, in addition to any one of the configuration, the workpiece sidegrip location designation unit may display, as an initial value, a statein which the end effector model is disposed to be directed downward, andthe workpiece model is disposed under the end effector model, in theworkpiece display region, and, in this state, may designate the gripposition at which the end effector model grips the workpiece model. Withthis configuration, when a grip position is registered for a workpiecemodel, an end effector model gripping the workpiece model is disposedover the workpiece model. Therefore, the end effector model is moveddownward in this state, and thus a user can easily intuitively recognizean operation of the end effector model gripping the workpiece model, andcan thus smoothly perform position registration work.

According to the robot setting apparatus related to an eighteenthaspect, in addition to any one of the configuration, the workpiece modelregistered by the workpiece model registration unit may be one of sixfundamental direction images in which the workpiece model is viewed frompositive and negative directions of each of a first axis, a second axis,and a third axis defining a virtual three-dimensional space andorthogonal to each other. With this configuration, instead of defining agrip attitude in an end effector in a state having a high degree offreedom for a workpiece model displayed at a three-dimensionally freeattitude, a grip position is set for a workpiece model viewed from axisdirections defining a virtual three-dimensional space, and can thus beset in a state in which an attitude of the workpiece model is defined tosome extent. Therefore, a user can more easily set the grip position ofthe workpiece model.

According to a nineteenth aspect, there is provided is a robot settingmethod of controlling a robot performing a bin picking operation of asensor unit measuring a three-dimensional shape of each of a pluralityof workpieces stacked in a work space and sequentially taking out theworkpieces with an end effector provided at a tip of an arm portion ofthe robot, the robot setting method including a step of displaying aworkpiece model virtually expressing a three-dimensional shape of aworkpiece with three-dimensional CAD data or a height image, and an endeffector model virtually expressing a three-dimensional shape of an endeffector with three-dimensional CAD data, in an image display regionrepresenting a virtual three-dimensional space; a step of designating agrip position at which the end effector model grips the workpiece modelfor the workpiece model displayed in the image display region in a statein which a grip reference point corresponding to a position at which theworkpiece model is gripped and a grip direction in which the endeffector model grips the workpiece model are defined for the endeffector model; and a step of automatically adjusting a relativeposition between the end effector model and the workpiece model suchthat the grip direction is orthogonal to a workpiece plane representingan attitude of the workpiece model displayed in the image displayregion, and the grip reference point and the grip position are locatedalong the grip direction. Consequently, in a case where a grip state inwhich a workpiece is gripped by an end effector is simulated, it ispossible to obtain an advantage that setting work for a position atwhich a workpiece model is gripped by an end effector model can beeasily performed.

According to a twentieth aspect, there is provided is a robot settingprogram for controlling a robot performing a bin picking operation of asensor unit measuring a three-dimensional shape of each of a pluralityof workpieces stacked in a work space and sequentially taking out theworkpieces with an end effector provided at a tip of an arm portion ofthe robot, the robot setting program causing a computer to realize afunction of displaying a workpiece model virtually expressing athree-dimensional shape of a workpiece with three-dimensional CAD dataor a height image, and an end effector model virtually expressing athree-dimensional shape of an end effector with three-dimensional CADdata, in an image display region representing a virtualthree-dimensional space; a function of defining a grip reference pointcorresponding to a position at which the workpiece is gripped for theend effector model; a function of defining a grip detection in which theend effector model grips the workpiece model; a function of designatinga grip position at which the end effector model grips the workpiecemodel for the workpiece model displayed in the image display region; anda function of automatically adjusting a relative position between theend effector model and the workpiece model such that the grip directionis orthogonal to a workpiece plane representing an attitude of theworkpiece model displayed in the image display region, and the gripreference point and the grip position are located along the gripdirection. With this configuration, in a case where a grip state inwhich a workpiece is gripped by an end effector is simulated, it ispossible to obtain an advantage that setting work for a position atwhich a workpiece model is gripped by an end effector model can beeasily performed.

According to a twenty-first aspect, there is provided a computerreadable recording medium recording the program or an apparatus storingthe program. The recording medium includes a magnetic disk, an opticaldisc, a magnetooptical disc, and a semiconductor memory, such as aCD-ROM, a CD-R, a CD-RW or a flexible disk, a magnetic tape, an MO, aDVD-ROM, a DVD-RAM, a DVD-R, DVD+R, a DVD-RW, a DVD+RW, a Blu-ray, andan HD DVD(AOD), and other media which can store programs. The programmay be stored in the recording medium so as to be distributed, and mayalso be downloaded via a network line such as the Internet so as to bedistributed. The apparatus storing the program includes a generalpurpose or dedicated apparatus in which the program is installed to beexecutable in the form of software or firmware. Each process or functionincluded in the program may be executed by computer executable programsoftware, and a process in each unit may be realized by hardware such asa predetermined gate array (an FPGA or an ASIC), or in a form in whichprogram software is mixed with a partial hardware module realizing apartial element of hardware.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating a state in which a binpicking operation is performed by using a robot system;

FIG. 2 is a block diagram of the robot system;

FIG. 3 is a perspective view illustrating an example of a sensor unit;

FIG. 4A is a schematic sectional view illustrating an example in whichworkpieces are put into a storage container and are stacked at random;

FIG. 4B is a schematic sectional view illustrating an example in whichworkpieces are stacked on a floor surface;

FIG. 4C is a perspective view illustrating a state in which workpiecesare arranged on a tray at a predetermined attitude;

FIG. 5A is a schematic diagram illustrating an example in which aworkpiece is gripped by an end effector;

FIG. 5B is a schematic diagram illustrating an example in which aworkpiece having a cavity is gripped from an inner surface of theworkpiece;

FIG. 5C is a schematic diagram illustrating an example of sucking andgripping a tabular workpiece;

FIG. 6 is a block diagram illustrating a robot system according toEmbodiment 1;

FIG. 7 is a perspective view illustrating a workpiece model built bythree-dimensional CAD data;

FIG. 8 is an image diagram illustrating a state in which the origin of asearch model is set for the workpiece model illustrated in FIG. 7;

FIG. 9A is a fundamental direction image in which the workpiece model inFIG. 7 is viewed from a positive direction of an X axis;

FIG. 9B is a fundamental direction image in which the workpiece model inFIG. 7 is viewed from a positive direction of a Y axis;

FIG. 9C is a fundamental direction image in which the workpiece model inFIG. 7 is viewed from a positive direction of a Z axis;

FIG. 9D is a fundamental direction image in which the workpiece model inFIG. 7 is viewed from a negative direction of the Z axis;

FIG. 10 is an image diagram illustrating a search model registrationscreen for registering a fundamental direction image as a search modelfor a three-dimensional search;

FIG. 11 is an image diagram illustrating actually measured data in whicha workpiece corresponding to the workpiece model in FIG. 7 is imagedfrom the positive direction of the X axis;

FIG. 12A is an image diagram illustrating a state in which featurepoints are extracted in the image in FIG. 9A viewed from the X axisdirection;

FIG. 12B is an image diagram illustrating a state in which the image inFIG. 12A is displayed in a three-dimensional manner;

FIG. 13A is an image diagram illustrating an input image in which aworkpiece group is displayed in a two-dimensional manner;

FIG. 13B is an image diagram illustrating an input image in which theimage in FIG. 13A is displayed in a three-dimensional manner;

FIG. 13C is an image diagram illustrating a state in which athree-dimensional search is performed on the image in FIG. 13A;

FIG. 13D is an image diagram illustrating a state in which the image inFIG. 13C is displayed in a three-dimensional manner;

FIG. 14 is an image diagram of a user interface screen indicating a gripregistration screen;

FIG. 15 is an image diagram illustrating a grip attitude adding screen;

FIG. 16A is an image diagram illustrating a state in which a pluralityof cubic workpieces are loaded in bulk;

FIG. 16B is an image diagram illustrating a state in which the workpiecemodel is registered in a search model;

FIG. 16C is an image diagram illustrating a state in which a gripattitude is registered in the workpiece model in FIG. 16B;

FIG. 17 is a functional block diagram illustrating a robot systemaccording to Embodiment 2;

FIG. 18 is an image diagram illustrating a state before an end effectormodel is rotated according to a Z-Y-X system Euler's angle;

FIG. 19 is an image diagram illustrating a state in which rotation isperformed by 90°about a Z axis in FIG. 18;

FIG. 20 is an image diagram illustrating a state in which rotation isperformed by 90°about a Y axis in FIG. 19;

FIG. 21 is an image diagram illustrating a state in which rotation isperformed by 90°about an X axis in FIG. 20;

FIG. 22 is an image diagram illustrating a rotational axis of R_(Y) orR_(Z) in the state illustrated in FIG. 21;

FIG. 23 is an image diagram illustrating a state in which a correctionrotational axis of R_(Z) is displayed in the state illustrated in FIG.22;

FIG. 24 is an image diagram illustrating a state in which a correctionrotational axis of R_(Y) is displayed in the state illustrated in FIG.22;

FIG. 25 is an image diagram illustrating a state in which a correctionrotational axis of R_(X) is displayed in the state illustrated in FIG.22;

FIG. 26 is a block diagram illustrating a robot system according toEmbodiment 3;

FIG. 27 is a flowchart illustrating procedures of teaching workperformed before an actual operation;

FIG. 28 is a flowchart illustrating procedures of registeringthree-dimensional CAD data as a search model;

FIG. 29 is a flowchart illustrating procedures of registering an endeffector model;

FIG. 30 is a flowchart illustrating procedures of registering an endeffector model to which an additional region is added;

FIG. 31 is a block diagram illustrating a robot system according toEmbodiment 4;

FIG. 32 is an image diagram illustrating an additional region settingscreen;

FIG. 33 is an image diagram illustrating a fundamental figure editingscreen;

FIG. 34 is a flowchart illustrating procedures of registering a gripposition and an attitude at which a workpiece is gripped by an endeffector;

FIG. 35A is an image diagram of a workpiece model formed ofthree-dimensional CAD data;

FIG. 35B is an image diagram illustrating an X-Y designation screen;

FIG. 36 is an image diagram illustrating a Z-R_(Z) designation screen;

FIG. 37 is an image diagram illustrating an R_(Y) designation screen;

FIG. 38 is an image diagram illustrating an R_(X) designation screen;

FIG. 39 is an image diagram illustrating a position parameterdesignation screen;

FIG. 40 is an image diagram illustrating an X-Y-Z designation screen;

FIG. 41 is an image diagram illustrating a state in which a gripposition is designated in a state in which a workpiece model in FIG. 35Bis projected from an inclined direction;

FIG. 42 is an image diagram illustrating a state in which a gripposition is designated on a three-dimensional image viewer;

FIG. 43 is an image diagram illustrating a tabular workpiece;

FIG. 44A is a plan view of a workpiece model indicating the workpiece inFIG. 43;

FIG. 44B is a bottom view of the workpiece model;

FIG. 44C is a front view of the workpiece model;

FIG. 44D is a rear view of the workpiece model;

FIG. 44E is a right side view of the workpiece model;

FIG. 44F is a left side view of the workpiece model;

FIG. 45 is an image diagram illustrating a result in which athree-dimensional search is performed on point group data obtained byimaging a workpiece group in which a plurality of the workpieces in FIG.43 are loaded in bulk;

FIG. 46 is a schematic diagram illustrating an example of expressing anattitude of a workpiece by using a Z-Y-X system Euler's angle;

FIG. 47 is an image diagram illustrating a search model registrationscreen;

FIG. 48 is a flowchart illustrating registration procedures for a searchmodel in which an attitude restriction is provided;

FIG. 49 is an image diagram illustrating an inclined angle/rotationangle setting screen;

FIG. 50A is an image diagram illustrating an attitude of a workpieceduring registration;

FIG. 50B is an image diagram illustrating an attitude of an input image;

FIG. 50C is an image diagram illustrating an inclined angle;

FIG. 51A is an image diagram illustrating an attitude of a workpieceduring registration;

FIG. 51B is an image diagram illustrating an attitude of an input image;

FIG. 51C is an image diagram illustrating a rotation angle;

FIG. 52 is a flowchart illustrating procedures of obtaining an inclinedangle and a rotation angle on the basis of a three-dimensional attitude;

FIG. 53A is an image diagram illustrating an attitude of a workpiecewhen a search model is registered;

FIG. 53B is an image diagram illustrating an attitude of an input image;

FIG. 53C is an image diagram illustrating a state in which an inclinedangle is obtained;

FIG. 54A is an image diagram illustrating a state in which a rotationalaxis is displayed in the search model in FIG. 53C;

FIG. 54B is an image diagram illustrating a state in which a Z′ axis inFIG. 54A is three-dimensionally rotated to match a Z axis;

FIG. 55A is an image diagram illustrating an attitude of a workpieceduring registration;

FIG. 55B is an image diagram illustrating a state in which aninclination in FIG. 54B is removed;

FIG. 55C is an image diagram illustrating a state in which a rotationangle when viewed from a Z vertical direction in FIG. 55B is obtained asa rotation angle;

FIG. 56 is an image diagram illustrating an end effector attachmentposition setting screen;

FIG. 57 is an image diagram illustrating a grip position designationscreen;

FIG. 58 is an image diagram illustrating a plural-grip positionselection screen;

FIG. 59 is an image diagram illustrating an end effector attachmentposition correction screen;

FIG. 60 is a flowchart illustrating procedures of automaticallycorrecting an end effector attachment position;

FIG. 61 is an image diagram illustrating an end effector imaging screen;

FIG. 62 is a flowchart illustrating procedures of registering actuallymeasured data as a search model;

FIG. 63 is a flowchart illustrating procedures of determining thepresence or absence of a grip solution for each workpiece during anactual operation;

FIG. 64 is a flowchart illustrating procedures during an actualoperation in which a picking operation is performed in a state in whichthe search model is registered according to the procedures in FIG. 48;

FIG. 65 is a flowchart illustrating procedures for interferencedetermination using a section model;

FIG. 66 is an image diagram illustrating an end effector model, andfundamental axes and section positions thereof;

FIGS. 67A to 67E are image diagrams illustrating sections in FIG. 66;

FIG. 68A is an image diagram illustrating a three-dimensional point andthe fundamental axes of the end effector model in FIG. 66;

FIG. 68B is an image diagram illustrating a state in which thethree-dimensional projection point in FIG. 68A does not interfere withthe section;

FIG. 68C is an image diagram illustrating a state in which thethree-dimensional point in FIG. 68A interferes with the section;

FIG. 69 is a flowchart illustrating procedures for interferencedetermination using an additional model;

FIG. 70 is a block diagram illustrating a robot system according toEmbodiment 7;

FIG. 71 is a flowchart illustrating procedures of determining thepresence or absence of a grip solution for each workpiece during anactual operation according to Embodiment 7;

FIG. 72 is a perspective view illustrating an example of a workpiece;

FIG. 73A is a height image in which the workpiece in FIG. 72 is viewedfrom a positive direction side of a Z axis;

FIG. 73B is a height image in which the workpiece in FIG. 72 is viewedfrom a negative direction side of the Z axis;

FIG. 73C is a height image in which the workpiece in FIG. 72 is viewedfrom a positive direction side of an X axis;

FIG. 73D is a height image in which the workpiece in FIG. 72 is viewedfrom a negative direction side of the X axis;

FIG. 73E is a height image in which the workpiece in FIG. 72 is viewedfrom a positive direction side of a Y axis;

FIG. 73F is a height image in which the workpiece in FIG. 72 is viewedfrom a negative direction side of the Y axis;

FIG. 74 is an image diagram illustrating a workpiece selection screen;

FIG. 75 is an image diagram illustrating a grip solution candidatedisplay screen in which good grip is selected;

FIG. 76 is an image diagram illustrating a grip solution candidatedisplay screen in which poor grip is selected;

FIG. 77 is an image diagram illustrating another example of a gripsolution candidate display screen in which poor grip is selected;

FIG. 78 is an image diagram illustrating an example of a grip solutioncandidate display screen for a search model;

FIG. 79 is an image diagram illustrating an example of a grip solutioncandidate display screen in which another grip position candidate isselected in FIG. 78;

FIG. 80 is an image diagram illustrating an example of a grip solutioncandidate display screen in which good grip is obtained by adding a gripposition;

FIG. 81 is a block diagram illustrating a robot system according toEmbodiment 8;

FIG. 82 is a flowchart illustrating procedures of registering a searchmodel according to Embodiment 8;

FIG. 83A is an image diagram illustrating a state in which a gripattitude is registered in a fundamental direction image in FIG. 73A;

FIG. 83B is an image diagram illustrating a state in which a gripattitude is registered in a fundamental direction image in FIG. 73B;

FIG. 83C is an image diagram illustrating a state in which a gripattitude is registered in a fundamental direction image in FIG. 73C;

FIG. 83D is an image diagram illustrating a state in which a gripattitude is registered in a fundamental direction image in FIG. 73D;

FIG. 83E is an image diagram illustrating a state in which a gripattitude is registered in a fundamental direction image in FIG. 73E;

FIG. 83F is an image diagram illustrating a state in which a gripattitude is registered in a fundamental direction image in FIG. 73F;

FIG. 84 is an image diagram illustrating a workpiece selection screen;

FIG. 85 is an image diagram illustrating a grip solution candidatedisplay screen in which a grip attitude is selected;

FIG. 86 is an image diagram illustrating a grip solution candidatedisplay screen in which a grip attitude is selected;

FIG. 87 is a flowchart illustrating procedures in which athree-dimensional search and grip propriety determination are performedduring an actual operation according to Embodiment 8;

FIG. 88 is a block diagram illustrating a robot system according toEmbodiment 9;

FIG. 89 is an image diagram illustrating a function selection screen;

FIG. 90 is an image diagram illustrating a search model registrationmethod selection screen;

FIG. 91 is an image diagram illustrating an actually measured dataimaging screen;

FIG. 92 is an image diagram illustrating a search model exclusion regionsetting screen;

FIG. 93 is an image diagram illustrating a rotation symmetry settingscreen;

FIG. 94 is an image diagram illustrating a search model registrationscreen;

FIG. 95 is an image diagram illustrating a search model registrationscreen to which a search model is added;

FIG. 96 is an image diagram illustrating a search region setting screen;

FIG. 97 is an image diagram illustrating a search region setting screenon which a floor surface designation dialog is displayed;

FIG. 98 is an image diagram illustrating a search region setting screenon which floor surface information is displayed;

FIG. 99 is an image diagram illustrating a search parameter settingscreen;

FIG. 100 is an image diagram illustrating a search parameter settingscreen on which a detection detail condition setting dialog isdisplayed;

FIG. 101 is an image diagram illustrating a setting screen for a searchparameter designated for each search model;

FIG. 102 is an image diagram illustrating a search model registrationmethod selection screen;

FIG. 103 is an image diagram illustrating a CAD data reading screen;

FIG. 104 is an image diagram illustrating a search model registrationscreen based on three-dimensional CAD data;

FIG. 105 is an image diagram illustrating a search model registrationscreen on which a rotation symmetry designation dialog is displayed;

FIG. 106 is an image diagram illustrating a model list display screen;

FIG. 107 is an image diagram illustrating a search result displayscreen;

FIG. 108 is an image diagram illustrating a search result displayscreen;

FIG. 109 is an image diagram illustrating a search result displayscreen;

FIG. 110 is an image diagram illustrating a search result displayscreen;

FIG. 111 is an image diagram illustrating a search result displayscreen;

FIG. 112 is an image diagram illustrating a search result displayscreen;

FIG. 113 is an image diagram illustrating a search result displayscreen;

FIG. 114 is an image diagram illustrating a model editing screen;

FIG. 115 is an image diagram illustrating a model region detail settingscreen;

FIG. 116 is an image diagram illustrating a feature detail settingscreen;

FIG. 117 is an image diagram illustrating a three-dimensional pickinginitial setting screen.

FIG. 118 is an image diagram illustrating an end effector model settingscreen;

FIG. 119 is an image diagram illustrating an end effector model editingscreen;

FIG. 120 is an image diagram illustrating a part adding screen;

FIG. 121 is an image diagram illustrating a CAD position and attitudesetting screen;

FIG. 122 is an image diagram illustrating a CAD position and attitudesetting screen;

FIG. 123 is an image diagram illustrating a CAD position and attitudesetting screen;

FIG. 124 is an image diagram illustrating an end effector model editingscreen;

FIG. 125 is an image diagram illustrating an additional region positionand attitude setting screen;

FIG. 126 is an image diagram illustrating an end effector model editingscreen;

FIG. 127 is an image diagram illustrating an end effector model editingscreen;

FIG. 128 is an image diagram illustrating an end effector model editingscreen;

FIG. 129 is an image diagram illustrating an end effector model editingscreen;

FIG. 130 is an image diagram illustrating an end effector model editingscreen;

FIG. 131 is an image diagram illustrating a grip registration screen;

FIG. 132 is an image diagram illustrating a grip setting dialog;

FIG. 133 is an image diagram illustrating a setting method selectiondialog;

FIG. 134 is an image diagram illustrating an X-Y designation screen;

FIG. 135 is an image diagram illustrating a Z-R_(Z) designation screen;

FIG. 136 is an image diagram illustrating a Z-R_(Z) designation screen;

FIG. 137 is an image diagram illustrating a Z-R_(Z) designation screen;

FIG. 138 is an image diagram illustrating an R_(Y) designation screen;

FIG. 139 is an image diagram illustrating an R_(Y) designation screen;

FIG. 140 is an image diagram illustrating an R_(X) designation screen;

FIG. 141 is an image diagram illustrating an R_(X) designation screen;

FIG. 142 is an image diagram illustrating a grip registration screen;

FIG. 143 is an image diagram illustrating a grip registration screen;

FIG. 144 is an image diagram illustrating a condition verificationscreen;

FIG. 145 is an image diagram illustrating a verification dialog;

FIG. 146 is an image diagram illustrating a detection result displaydialog;

FIG. 147 is an image diagram illustrating a detection result displaydialog;

FIG. 148 is an image diagram illustrating a detection condition detailsetting screen;

FIG. 149 is an image diagram illustrating a place setting screen;

FIG. 150 is a flowchart illustrating procedures of manually performingdeviation correction;

FIG. 151 is an image diagram illustrating an example of a deviationcorrection screen in a case where there is deviation which cannot beignored;

FIG. 152 is an image diagram illustrating an example of a deviationcorrection screen in a case where there is deviation which can beignored; and

FIG. 153 is an image diagram illustrating an example of a deviationcorrection screen having functions of manually and automaticallycorrecting deviation.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention will be described withreference to the drawings. However, the embodiments described below areonly examples for embodying the technical spirit of the presentinvention, and the present invention is not limited to the followingembodiments. In the present specification, a member recited in theclaims is not specified to a member in the embodiments. Particularly,regarding dimensions, materials, shapes, relative arrangements, and thelike of the constituent components described in the embodiments, thescope of the present invention is not intended to be limited theretounless otherwise specifically stated, and they are only explanationexamples. Sizes of members or positional relationships therebetweenillustrated in each drawing may be exaggerated for better understandingof description. In the following description, the same name andreference numeral indicate the same or equivalent member, and thus adetailed description thereof will be omitted as appropriate. Eachelement forming the present invention may have an aspect in which aplurality of elements form the same member, and a single member is usedas the plurality of elements, and, on the contrary, a function of asingle member may be shared and realized by a plurality of members.

Embodiment 1

FIG. 1 illustrates a configuration example of a robot system 1000 forpicking a picking target workpiece as Embodiment 1. In this example, adescription will be made of an example of bin picking in which aplurality of workpieces WK stacked in a work space are sequentiallytaken out by using a robot RBT, and are disposed at a predeterminedposition. The robot RBT includes an arm portion ARM also referred to asa manipulator, and an end effector EET provided at a tip of the armportion ARM. The arm portion ARM is provided with a plurality of movableportions, and the end effector EET is moved to a desired position on thebasis of an angle formed between two arms or through rotation of an armfulcrum. The end effector EET can grip or release the workpiece WK.

An operation of the robot RBT is controlled by a robot controller 6. Therobot controller 6 controls movement of the arm portion ARM or openingand closing operations of the end effector EET. The robot controller 6acquires information required for control of the robot RBT from a robotsetting apparatus 100. For example, three-dimensional shapes of theworkpieces WK which are a plurality of parts randomly put in a storagecontainer BX are acquired by a sensor unit 2 such as a three-dimensionalcamera or a light, and a calculation unit 10 of the robot settingapparatus 100 detects a position or an attitude of the workpiece, andsends information to the robot controller 6. The robot controller 6causes the end effector EET provided at the tip of the arm portion ARMof the robot RBT to grip the workpieces WK one by one, and arranges theworkpieces WK at a predetermined position on a stage STG, for example,on a conveyer belt.

FIG. 2 is a functional block diagram of the robot system 1000. The robotsystem 1000 illustrated in FIG. 2 includes the robot setting apparatus100, the sensor unit 2, a display unit 3, an operation unit 4, a robotmain body 5, the robot controller 6, and a robot operation tool 7.

The operation unit 4 performs setting regarding image processing. Thesensor unit 2 acquires a three-dimensional shape obtained by imaging aworkpiece. Setting or an operation state is checked by using the displayunit 3. The calculation unit 10 performs a three-dimensional search orinterference determination, and calculates a grip solution or the like.On the other hand, the robot controller 6 controls the robot accordingto a result in the calculation unit 10. The robot operation tool 7 setsan operation of the robot. In the example illustrated in FIG. 2, theoperation unit 4 and the robot operation tool 7 are separate members,but may be integrated into a single member.

The sensor unit 2 is a member which is called a robot vision, and imagesa work space or a workpiece. Three-dimensional shape data indicating athree-dimensional shape of each of workpieces loaded in bulk is obtainedon the basis of an image captured by the sensor unit 2. Methods ofacquiring a three-dimensional shape include a pattern projection method,a stereo method, a lens focusing method, a light section method, anoptical radar method, an interference method, a TOF method, and thelike. In the present embodiment, a phase shift method in the patternprojection method is used.

A configuration of the sensor unit 2 is determined depending on atechnique of measuring a three-dimensional shape. The sensor unit 2includes a camera, a light, or a projector. For example, in a case wherea three-dimensional shape of a workpiece is measured according to thephase shift method, the sensor unit 2 includes a projector PRJ, and aplurality of cameras CME1, CME2, CME3 and CME4 as illustrated in FIG. 3.The sensor unit is formed of a plurality of members such as cameras orprojectors, and may also be formed by integrating the members with eachother. For example, a 3D imaging head having a head shape by integratingcameras or projectors with each other may be used as the sensor unit.

The three-dimensional shape data may be generated on the sensor unitside. In this case, an image processing IC or the like realizing afunction of generating three-dimensional shape data is provided on thesensor unit side. Alternatively, there may be a configuration in whichthree-dimensional shape data is not generated by the robot settingapparatus side, and the robot setting apparatus performs imageprocessing on an image captured by the sensor unit side so as togenerate three-dimensional shape data such as a three-dimensional image.

Calibration which will be described later may be performed on the basisof an image captured by the sensor unit 2 such that an actual positioncoordinate (a coordinate of a movement position of the end effector EET)of the workpiece WK can be linked to a position coordinate on an imagedisplayed on the display unit 3.

The robot setting apparatus 100 performs a three-dimensional search,interference determination, grip solution calculation, and the like onthe basis of the three-dimensional shape data of the workpiece obtainedin the above-described way. As the robot setting apparatus 100, ageneral purpose computer in which a dedicated robot setting program isinstalled, or an image processing controller or a robot vision apparatusspecially designed may be used. The example illustrated in FIG. 2 is anexample in which the sensor unit 2 or the robot controller 6 isconfigured separately from the robot setting apparatus 100, but thepresent invention is not limited to this configuration, and, forexample, the sensor unit and the robot setting apparatus may beintegrally formed, or the robot controller may be incorporated into therobot setting apparatus. As mentioned above, division of the membersillustrated in FIG. 2 is only an example, and a plurality of members maybe integrated with each other. For example, the operation unit 4operating the robot setting apparatus 100 and the robot operation tool 7operating the robot controller 6 may be integrated into a single member.

However, the sensor unit 2 is provided separately from the robot mainbody 5. In other words, the present invention is applied to a formcalled an off-hand type in which the sensor unit 2 is not provided inthe arm portion ARM of the robot main body 5. In other words, thepresent invention does not include a form called an on-hand type inwhich the sensor unit is provided in the end effector.

The display unit 3 is a member for displaying a three-dimensional shapeof a workpiece acquired in the robot setting apparatus 100 or checkingvarious settings or an operation state, and may employ a liquid crystalmonitor, an organic EL display, or a CRT. The operation unit 4 is amember for performing various settings such as image processing, and mayemploy an input device such as a keyboard or a mouse. The display unit 3is formed of a touch panel, and thus the operation unit and the displayunit can be integrally formed.

For example, in a case where the robot setting apparatus 100 is formedof a computer in which a robot setting program is installed, a graphicaluser interface (GUI) screen of the robot setting program is displayed onthe display unit 3. Various settings can be performed on the GUIdisplayed on the display unit 3, and a processing result such as asimulation result can be displayed. In this case, the display unit 3 maybe used as a setting unit for performing various settings.

The robot controller 6 controls an operation of the robot on the basisof information obtained through imaging in the sensor unit 2. The robotoperation tool 7 is a member for setting an operation of the robot mainbody 5, and may employ a pendant or the like.

The robot main body 5 includes the movable arm portion ARM and the endeffector EET fixed at the tip of the arm portion ARM. The robot mainbody 5 is driven by the robot controller 6, picks a single workpiece WKby operating the arm portion ARM, moves the workpiece WK to a desiredposition, and places and releases the workpiece WK at the position. Todo so, the end effector EET gripping the workpiece WK is provided at thetip of the arm portion ARM. A placement position at which the workpieceWK is placed may be, for example, a position on a tray or a position ona conveyer.

As illustrated in FIG. 1, a plurality of workpieces WK are stored in thestorage container BX such as a returnable box at random. The sensor unit2 is disposed over such a work space. The sensor unit 2 includes acamera or a light, and the sensor unit 2 can measure a three-dimensionalshape of the workpiece WK. The robot controller 6 specifies a griptarget workpiece WK from among the plurality of workpieces on the basisof the three-dimensional shape of the workpiece WK measured by thesensor unit 2, and controls the robot to grip the workpiece WK. The armportion ARM is operated to be moved to a predefined placement position,for example, a predetermined position on the stage STG, in a state ofgripping the workpiece WK, and places the workpiece WK at apredetermined attitude. In other words, the robot controller 6 specifiesa picking target workpiece WK with the sensor unit 2, and controls anoperation of the robot such that the end effector EET grips theworkpiece WK, and places the workpiece WK at a placement position at apredetermined reference attitude, and then the end effector EET isopened.

Here, in the present specification, the term “bin picking” indicates notonly an example in which the workpieces WK put in the storage containerBX as illustrated in FIG. 4A and randomly stacked are gripped by therobot and are placed at a predetermined position, but also an example inwhich the workpieces WK stacked in a predetermined region without usinga storage container are gripped and placed as illustrated in FIG. 4B, oran example in which the workpieces WK arranged and stacked at apredetermined attitude are sequentially gripped and placed asillustrated in FIG. 4C. A state in which workpieces are stacked in anoverlapping manner is not necessary, and a state in which workpieces arerandomly put on a plane without the workpieces overlapping each other isalso referred to as a bulk state (this is for the same reason as thereason why even a case where workpieces are sequentially picked and donot overlap each other at the end of picking is still referred to as binpicking).

In the example illustrated in FIG. 1, the sensor unit 2 is fixed overthe work space, but may be disposed at any constant position such as anoblique position, a side position, or a lower position as long as thework space can be imaged at the position. However, an aspect is excludedin which the sensor unit is disposed on a movable position which is notfixed such as an upper part of the arm portion ARM. The number ofcameras or lights of the sensor unit 2 is not limited to one, and may beplural. Connection between the sensor unit 2 or the robot and the robotcontroller 6 is not limited to wired connection, and may be wirelessconnection.

The term “gripping of a workpiece” includes not only an example in whichan outer surface of the workpiece WK is clamped as illustrated in FIG.5A, but also an example in which claws of an end effector EET2 areinserted into the inside of a workpiece WK2 having a cavity and areexpanded, and thus the workpiece is held, as illustrated in FIG. 5B, andan example in which a tabular workpiece WK3 is sucked and held by an endeffector EET3 as illustrated in FIG. 5C. Hereinafter, a description willbe made of an aspect of catching an outer surface of a workpiece fromboth sides as an example of gripping of the workpiece. Hereinafter, adescription will be made of setting of a grip position (teaching work)in a bin picking operation in which, as illustrated in FIG. 1, aplurality of workpieces are stored and stacked at random in the storagecontainer BX, the end effector EET grips the workpieces WK one by one inthis state, and repeatedly performs work of placing the workpiece WK ata placement position.

When the robot system 1000 performs the bin picking operation, teachingincluding setting for performing the bin picking operation is performedin advance. Specifically, a relationship between a part of a workpieceto be gripped and an attitude of the end effector, a grip position, andthe like are registered. Such setting is performed by the robotoperation tool 7 such as a pendant.

Teaching Work

FIG. 6 is a functional block diagram of the robot system including therobot setting apparatus realizing a function of teaching the gripposition. The robot system 1000 illustrated in FIG. 6 includes the robotsetting apparatus 100, the display unit 3, the operation unit 4, thesensor unit 2, and the robot RBT.

The sensor unit 2 three-dimensionally measures a three-dimensional shapeof a workpiece disposed at a work position. The sensor unit 2 iscontrolled by a sensor control unit 2 b. In this example, the sensorcontrol unit 2 b is integrally formed with the sensor unit 2, but may beprovided separately therefrom. The robot includes the arm portion ARMand the end effector EET. The robot is controlled by the robot settingapparatus 100, and grips a workpiece at a grip position. Herein, a statein which a workpiece is gripped at a reference attitude and is placed atthe reference attitude is imaged by the sensor unit 2 so as to beregistered. Here, the reference attitude includes a position and anattitude of a workpiece.

Display Unit 3

The display unit 3 displays a workpiece model which virtually expressesa three-dimensional shape of a workpiece or an end effector model whichvirtually expresses a three-dimensional shape of an end effector and isformed of three-dimensional CAD data, in a three-dimensional shape in avirtual three-dimensional space. A height image is an image havingheight information, and is also referred to as a distance image or thelike. The display unit 3 has a six-drawing display region 3 a in which afundamental direction image of a workpiece model is displayed as sixdrawings. Consequently, each attitude of a workpiece model is displayedin six drawings, grip position setting work can be performed on afundamental direction image in which a grip position is easily set, andthus it is possible to easily perform grip position setting work whichis troublesome in the related art.

The operation unit 4 is a member for performing various settings such asimage processing, and may employ an input device such as a keyboard or amouse. The display unit 3 is formed of a touch panel, and thus theoperation unit and the display unit can be integrally formed.

Robot Setting Apparatus 100

The robot setting apparatus 100 in FIG. 6 includes an input imageacquisition unit 2 c, a storage unit 9, the calculation unit 10, aninput/output interface 4 b, a display interface 3 f, and a robotinterface 6 b.

The input image acquisition unit 2 c acquires an input image including athree-dimensional shape on the basis of an image including a pluralityof workpieces and peripheral objects measured by the sensor unit 2. Theinput image including a three-dimensional shape may be generated on thesensor unit side or the sensor control unit side, or may be generated onthe robot setting apparatus side (for example, the input imageacquisition unit). In the example illustrated in FIG. 6, an inputinterface is provided such that the robot setting apparatus 100 acquiresan input image exhibiting a three-dimensional shape acquired in thesensor unit 2. However, this configuration is only an example, and aninput image which is captured in advance and is held in a storage unitsuch as a recording medium may be read and acquired.

The storage unit 9 is a member for holding various settings, and mayemploy a nonvolatile memory, a hard disk, a storage medium, or the like.The storage unit 9 functions as a grip position preservation unitpreserving a grip position of a workpiece model or an end effectormodel.

The input/output interface 4 b is connected to an input device such as akeyboard or a mouse, and receives input of data.

The display interface 3 f forms an output interface with the displayunit, and is controlled to display image data which is generated by thecalculation unit 10 and is displayed on the display unit.

The robot interface 6 b forms a communication interface with the robotcontroller 6.

Calculation Unit 10

The calculation unit 10 includes a positioning portion 8 c, afundamental direction image generation portion 8 e′, a fundamentaldirection image selection portion 8 e, a search model selection portion8 i, a workpiece model registration portion 8 t, a search modelregistration portion 8 g, an end effector model registration portion 8u, a grip position specifying portion 8 d, an end effector attachmentsurface setting portion 8 f, a rotation angle restriction portion 8 h, athree-dimensional search portion 8 k, and a three-dimensional pickdetermination portion 8 l.

The positioning portion 8 c is a member for adjusting a position and anattitude of a workpiece model displayed on the display unit 3 in avirtual three-dimensional space.

The fundamental direction image generation portion 8 e′ is a member forgenerating at least three height images in which the workpiece modelpositioned in the virtual three-dimensional space by the positioningportion 8 c is viewed from respective axis directions of three axeswhich are orthogonal to each other in the virtual three-dimensionalspace as fundamental direction images. Since fundamental directionimages are automatically generated as mentioned above, and a user is notrequired to separately acquire fundamental direction images by manuallychanging a direction of a workpiece, it is possible to obtain anadvantage of being capable of achieving labor-saving for grip positionregistration work.

The fundamental direction image selection portion 8 e is a member forselecting one of a plurality of fundamental direction images which aredifferent from fundamental direction images of which viewing ways aredifferent from each other with respect to at least three fundamentaldirection images displayed on the display unit 3. As mentioned above, aface having the same viewing way is deleted, display or the like of anunnecessary fundamental direction image is excluded, and thus it ispossible to further simplify setting work. For example, one offundamental direction images including a top face and a bottom face of acylindrical workpiece having the same appearance is deleted. Thefundamental direction image selection portion 8 e may allow a user tomanually select one of at least three fundamental direction images in astate in which the fundamental direction images are displayed on thedisplay unit 3. Alternatively, the fundamental direction image selectionportion may automatically extract fundamental direction images of whichviewing ways are the same as each other from at least three fundamentaldirection images, and select one thereof.

The search model selection portion 8 i is a member for selecting afundamental direction image to be registered as a search model. As willbe described later, a search model used for a three-dimensional searchand a model for specifying a grip position are commonized, and thus thesearch model selection portion 8 i and the fundamental direction imageselection portion 8 e can be formed as a common image selection portion8 j.

The workpiece model registration portion 8 t is a member for registeringa workpiece model virtually expressing a three-dimensional shape of aworkpiece. Herein, for example, the workpiece model registration portion8 t uses three-dimensional point group data obtained by imaging a realworkpiece as a workpiece model. In this case, three-dimensional pointgroup data acquired by the sensor unit 2 or the input image acquisitionunit 2 c is registered as a workpiece model by the workpiece modelregistration portion 8 t. Alternatively, three-dimensional CAD dataindicating a shape of a workpiece which is separately created is readand registered as a workpiece model. In this case, three-dimensional CADdata which is input via the input/output interface 4 b is registered asa workpiece model by the workpiece model registration portion 8 t.Alternatively, three-dimensional CAD data simulating a workpiece may becreated and registered as a workpiece model. In this case, the workpiecemodel registration portion 8 t realizes a function of simplethree-dimensional CAD.

Search Model Registration Portion 8 g

The search model registration portion 8 g is a member for registering asearch model virtually expressing a three-dimensional shape of aworkpiece and used to perform a three-dimensional search for specifyingan attitude and a position of each workpiece with respect to a pluralityof workpiece groups included in an input image acquired by the inputimage acquisition unit 2 c. Since the search model registration portion8 g is provided, a search model used to perform a three-dimensionalsearch is registered in common to a fundamental direction image fordesignating a grip position of a workpiece model, and thus a user canachieve labor-saving for setting work. A grip position of a workpiece isalso registered for each fundamental direction image used to search fora workpiece which can be gripped during an actual operation. Therefore,it is possible to prevent a fundamental direction image in which thereis no grip position from being wastefully searched, and, conversely, itcan be examined whether or not grip is possible at a grip position setin a searched fundamental direction image. Consequently, it is possibleto perform a process with high efficiency.

The search model registration portion 8 g is preferably configured toselect whether or not a fundamental direction image selected by thefundamental direction image selection portion 8 e is used as a searchmodel for a three-dimensional search. Consequently, it is possible toselect whether or not a fundamental direction image is used as a searchmodel as a three-dimensional search, in other words, an unnecessaryfundamental direction image can be excluded from a three-dimensionalsearch target, for example, a fundamental direction image which may bewrongly detected as an image in which a tabular workpiece is viewed froma side surface is excluded. Therefore, a state in which the tabularworkpiece is upright can be set not to undergo a three-dimensionalsearch, and thus a restriction on an attitude of a workpiece model canbe substantially easily set.

The search model registration portion 8 g and the workpiece modelregistration portion 8 t may be separately provided, and may beintegrally provided. For example, in the robot system 1000 in FIG. 6,the search model registration portion 8 g and the workpiece modelregistration portion 8 t are integrated into a common model registrationportion 8 g′. Consequently, a model regarding a single workpiece isregistered, and can thus be used in common to registration of a gripposition and registration of a search model for a three-dimensionalsearch, and thus it is possible to obtain an advantage that setting canbe simplified.

The end effector model registration portion 8 u is a member forregistering an end effector model which is three-dimensional CAD dataand virtually expresses a three-dimensional shape of an end effector.

The end effector attachment surface setting portion 8 f is a member fordisplaying an end effector model and an attachment surface used toattach the end effector model to the tip of the arm portion of the roboton the display unit 3, and defining an attitude of the end effectormodel for the attachment surface in a state in which the attachmentsurface is displayed on the display unit 3.

The rotation angle restriction portion 8 h is a member for setting arange of a rotation angle which is allowed for rotation of eachworkpiece model for each search model selected by the search modelregistration portion 8 g which registers one of fundamental directionimages as a search model for performing a three-dimensional search forspecifying an attitude and a position of each workpiece with respect toa plurality of workpiece groups loaded in bulk.

Grip Position Specifying Portion 8 d

The grip position specifying portion 8 d is a member for specifying agrip position at which a workpiece model is gripped by an end effectorwith respect to at least one of height images in a state in which atleast three fundamental direction images are displayed on the displayunit 3, the fundamental direction images being images in which theworkpiece model positioned in the virtual three-dimensional space by thepositioning portion 8 c is viewed from axis directions of three axeswhich are orthogonal to each other in a virtual three-dimensional space.The grip position specifying portion 8 d includes a workpiece side griplocation designation portion 8 d 1, an end effector side grip settingportion 8 d 2, and a relative position setting portion 8 d 5.

The workpiece side grip location designation portion 8 d 1 is a memberfor designating a grip position at which a workpiece model indicated bya fundamental direction image is gripped by an end effector model withrespect to the fundamental direction image in a state in which aplurality of fundamental direction images selected by the fundamentaldirection image selection portion 8 e are displayed on the display unit3. The workpiece side grip location designation portion 8 d 1 isconfigured to register a plurality of grip positions with respect toeach of a plurality of fundamental direction images.

End Effector Side Grip Setting Portion 8 d 2

The end effector side grip setting portion 8 d 2 is a member forperforming setting regarding grip of a workpiece model for an endeffector model. The end effector side grip setting portion 8 d 2 mayinclude a grip reference point setting portion 8 d 3 and a gripdirection setting portion 8 d 4. The grip reference point settingportion 8 d 3 defines a grip reference point corresponding to a positionwhere a workpiece model is gripped with respect to an end effectormodel. On the other hand, the grip direction setting portion 8 d 4defines a grip direction in which a workpiece model is gripped by an endeffector model. The grip reference point setting portion 8 d 3 and thegrip direction setting portion 8 d 4 may be integrally formed, and maybe separately formed. The grip reference point setting portion 8 d 3 andthe grip direction setting portion 8 d 4 may use predetermined valuesset in advance as a grip reference point and a grip direction. Forexample, the grip reference point is set to the center between clawswith a workpiece interposed therebetween, provided at a tip of an endeffector model. The grip direction is set to any one of tool coordinateaxes for defining an end effector model. For example, if a Z axisdirection is used as the grip direction, an operation is performed inwhich an end effector model is moved along the Z axis direction, thatis, a height direction, so as to approach and grip a workpiece model,and thus a user can easily perceive the operation. Alternatively, in thegrip reference point setting portion 8 d 3 or the grip direction settingportion 8 d 4, a grip reference point or a grip direction may beadjusted by a user.

Relative Position Setting Portion 8 d 5

The relative position setting portion 8 d 5 is a member for moving anend effector model displayed on the display unit 3 until the endeffector model interferes with a workpiece model, and automaticallydefining a grip state at an attitude separated from a position reachingthe interference position by a predetermined distance. Consequently, itis possible to indicate a grip position by automatically moving andbringing an end effector model into contact with a workpiece instead ofa user manually moving and bringing the end effector model into contactwith the workpiece, and thus to obtain an advantage that considerablelabor-saving can be achieved for work required to be performed by a userside.

Registration of Grip Position

In teaching work, a positional relationship between a workpiece duringgripping of the workpiece and an end effector is registered as a gripposition. Hereinafter, a description will be made of an example in whicha workpiece is gripped by using a hand portion as an end effector as arepresentative example of grip. During an actual operation of robotpicking using a robot vision in a state in which a grip position isregistered, each workpiece is detected by the robot vision from aworkpiece group in which a plurality of workpieces are loaded in bulk, agrip position of an end effector side is computed with respect to aposition or an attitude of the detected workpiece, and the robot isoperated to pick the workpiece such that the end effector is located atthe computed position. Here, as a method of registering a grip position,there is a method of registering a grip position by actually operating arobot, or a method of registering a grip position by operating an endeffector model in a virtual three-dimensional space using athree-dimensional CAD. However, in the method of registering a gripposition by actually moving a robot, there is a problem in thattime-consuming registration work is required to be performed by actuallymoving a robot, and a large-scale verification environment is necessaryor time is required for trial and error. On the other hand, in themethod of registering a grip position by virtually operating a robot ona three-dimensional CAD space, there is an advantage in that a gripposition can be registered without an actual robot, but athree-dimensional attitude of a virtual end effector model is requiredto be accurately aligned with a three-dimensional attitude of a virtualworkpiece model, and thus setting such as positioning ofthree-dimensional coordinates is difficult. This method requiresthree-dimensional CAD data of a workpiece and an end effector, and thussetting cannot be performed in a state in which three-dimensional CADdata is not available.

Therefore, in the present embodiment, a plurality of height imagesviewed from respective axis directions of a three-dimensional CAD modelare displayed as fundamental direction images, and, a fundamentaldirection image desired by a user is selected from among the fundamentaldirection images, a grip position is set for the selected fundamentaldirection image, and thus grip registration on a virtualthree-dimensional space can be easily performed. As a result, when agrip position is registered by operating an end effector model on avirtual three-dimensional space in which a workpiece model ofthree-dimensional CAD data simulating a workpiece is disposed, gripregistration is performed by selecting a fundamental direction imageviewed from each axis direction for defining the virtualthree-dimensional space, and thus an attitude of the end effector modelviewed from a substantially vertical direction can be registered foreach fundamental direction image. Therefore, grip setting can be easilyperformed. According to this method, even in a case where there is nothree-dimensional CAD data, data obtained by three-dimensionallymeasuring a real workpiece can be used as a fundamental direction image.Thus, even in a case where there is no three-dimensional CAD data, gripregistration can be easily performed on a virtual three-dimensionalspace according to the same procedures.

Three-Dimensional Pick Determination Portion 8 l

The three-dimensional pick determination portion 8 l is a member foradjusting a relative position between an end effector model and aworkpiece model such that a grip direction defined by the grip directionsetting portion 8 d 4 is orthogonal to a workpiece plane representing anattitude of the workpiece model displayed in the image display region,and a grip reference point and a grip position are located along thegrip direction. Consequently, in a case where a grip state in which aworkpiece is gripped by an end effector is simulated, it is possible toobtain an advantage that setting work for a position at which aworkpiece model is gripped by an end effector model can be easilyperformed. Particularly, since a grip direction is orthogonal to aworkpiece plane, and a grip reference point and a grip position arelocated on an axis in the grip direction, an end effector model has onlyto come close to a workpiece model along the grip direction such thatthe grip position can be adjusted, and thus it is possible to obtain anadvantage that a work burden on a user side is considerably reduced. Inthe related art, a user performs work for alignment with an attitude ofgripping a workpiece with the naked eyes while manually moving an endeffector, and is thus considerably troublesome work since the number ofparameters is large, and a degree of freedom increases depending onadjustment of a position or an attitude of the end effector. Incontrast, a grip reference position and a grip direction are defined onan end effector model side in advance, and the grip direction is set tobe orthogonal to a workpiece plane, and a grip reference point of theend effector model and a grip position of a workpiece are set to belocated on an axis in the grip direction. Thus, a movement direction ofthe end effector model is defined, and a user can obtain a grip state byadjusting only a distance between the end effector model and a workpiecemodel. As a result, considerable labor-saving for fitting work betweengrip positions of an end effector model and a workpiece model, which istroublesome work in the related art, can be expected.

The three-dimensional pick determination portion 8 l preferablyautomatically performs the work of adjusting a relative position betweenan end effector model and a workpiece model such that a grip directionis orthogonal to a workpiece plane and a grip reference point and a gripposition are located on an axis in the grip direction. Consequently,automatic adjustment is performed such that a grip reference positionand a grip direction are defined on an end effector model side inadvance, and the grip direction is set to be orthogonal to a workpieceplane, and a grip reference point of the end effector model and a gripposition of a workpiece are set to be located on an axis in the gripdirection. Thus, a movement direction of the end effector model isdefined, and a user can obtain a grip state by adjusting only a distancebetween the end effector model and a workpiece model. As a result,considerable labor-saving for fitting work between grip positions of anend effector model and a workpiece model, which is troublesome work inthe related art, can be expected.

However, the present invention is not limited to automatic adjustment inthe three-dimensional pick determination portion, and, for example,there may be a configuration in which, when a user manually performsadjustment such that a grip direction is orthogonal to a workpieceplane, and a grip reference point of an end effector model and a gripposition of a workpiece are located on an axis in the grip direction inthe positioning portion 8 c, the three-dimensional pick determinationportion supports the adjustment work. For example, there may be aconfiguration in which, in a first stage, in a state in which aworkpiece plane and a grip direction are displayed in an image displayregion, an auxiliary line is displayed to a user such that the workpieceplane and the grip direction are orthogonal to each other, or text or animage including the content that “adjust an end effector model such thatthe grip direction is orthogonal to the workpiece plane” is displayed toprompt the user to perform adjustment work.

In a second stage, the three-dimensional pick determination portion mayprompt the user to perform setting by displaying an extension line of agrip direction in the image display region such that a grip referencepoint of the end effector model and a grip position of the workpiece arelocated on an axis in the grip direction, and displaying a message that“adjust the end effector model such that a grip reference point of theend effector model and a grip position of the workpiece are located onthe axis in the grip direction”.

The three-dimensional pick determination portion 8 l may realize afitting function. For example, the three-dimensional pick determinationportion 8 l may move an end effector model along a grip direction untilthe end effector model interferes with a workpiece model, and mayautomatically define a grip state at an attitude separated from aposition reaching the interference position by a predetermined distancein a state in which a grip direction is orthogonal to a workpiece planeand a grip reference point and a grip position are located on an axis inthe grip direction by adjusting a relative position between the endeffector model and the workpiece model. Consequently, it is possible toobtain an advantage that automatic adjustment to a position and anattitude at which a workpiece model is gripped by an end effector modelcan be performed, and thus a work burden on a user side can be furtherreduced.

Three-Dimensional Search Portion

The three-dimensional search portion is a member for performing athree-dimensional search for specifying an attitude and a position ofeach workpiece from an input image by using a search model registered bythe search model registration portion. Prior to a three-dimensionalsearch, in the search model registration portion, a search modelvirtually expressing a three-dimensional shape of a workpiece isregistered in advance so as to be used to perform a three-dimensionalsearch for specifying an attitude and a position of each workpiece froman input image indicating a state in which a plurality of workpiecegroups are loaded in bulk. In this state, the three-dimensional pickdetermination portion 8 l determines whether or not the workpiece can begripped by the end effector at a grip position designated for theworkpiece by the workpiece side grip location designation portion on thebasis of a search result of the input image searched by thethree-dimensional search portion. For example, the input imageacquisition unit acquires an input image including a three-dimensionalshape of each workpiece from among images of a plurality of workpiecegroups measured by the sensor unit, and the three-dimensional searchportion performs a three-dimensional search for specifying an attitudeand a position of each workpiece from the input image by using a searchmodel registered by the search model registration portion. Consequently,a three-dimensional search can be performed from an input image acquiredby actually imaging a workpiece, and thus it is possible to perform gripdetermination conforming more to the actual circumstances.

Search Model

In bin picking, first, each workpiece is required to be extracted from aplurality of workpiece groups loaded in bulk in order to determine aworkpiece which can be gripped. Here, a shape of a search targetworkpiece is registered as a workpiece model in advance with respect toshapes of a workpiece group having height information obtained by thesensor unit, and a three-dimensional search is performed by using theworkpiece model such that a position and an attitude of each workpieceare detected.

Fundamental Direction Image

A search model used to perform a three-dimensional search for aworkpiece is created by using a height image in which a workpiece isviewed from a specific direction. A height image used as a search modelmay use three-dimensional CAD data which is a workpiece modelthree-dimensionally expressing a workpiece, or actually measured dataobtained by actually imaging a workpiece in the sensor unit. Herein, adescription will be made of an example in which three-dimensional CADdata is registered as a search model. For example, as illustrated inFIG. 7, a description will be made of a case where a workpiece model CWMbuilt by using three-dimensional CAD data is used as a search model. Sixheight images, that is, six drawings viewed from positive directions andnegative directions of respective axis directions of three axes (forexample, an X axis, a Y axis, and a Z axis) which are orthogonal to eachother on a virtual three-dimensional space are acquired as fundamentaldirection images on the basis of the workpiece model CWM. For example,the fundamental direction image generation portion 8 e′ in FIG. 6generates six fundamental direction images, and displays the fundamentaldirection images in the six-drawing display region 3 a of the displayunit 3. The six fundamental direction images are automaticallycalculated to be obtained as “top”, “bottom”, “left”, “right”, “front”,and “rear” of the workpiece model CWM, that is, a plan view, a bottomview, a left side view, a right side view, a front view, and a rearview, on the basis of the origin (which will be described later indetail) of the workpiece model CWM or faces forming the workpiece modelCWM by the fundamental direction image generation portion 8 e′. Here,the “top” indicates a height image viewed from a positive direction(positive side) of the Z axis, the “bottom” indicates a height imageviewed from a negative direction (negative side) of the Z axis, the“left” indicates a height image viewed from a negative direction of theX axis, the “right” indicates a height image viewed from a positivedirection of the X axis, the “front” indicates a height image viewedfrom a negative direction of the Y axis, and the “rear” indicates aheight image viewed from a positive direction of the Y axis. However,these are only examples, and different coordinate systems may be used.For example, a straight line of X=Y in an X-Y plane may be set as anaxis, and height images viewed from positive and negative directions ofrespective axes based on coordinate systems orthogonal to the axis maybe used. When height images are generated on the basis ofthree-dimensional CAD data, the height images are not necessarily heightimages viewed from directions (“top”, “bottom”, “left”, “right”,“front”, and “rear”) orthogonal to an axis of CAD, and, for example, anattitude (viewpoint) of a workpiece model may be arbitrarily changed,and height images from changed viewpoints may be generated.

Origin of Workpiece Model

Here, the origin of a workpiece model is automatically determined by therobot setting apparatus on the basis of coordinate information ofthree-dimensional CAD data. For example, a virtual cuboid IBXcircumscribing the workpiece model CWM is defined as indicated by dashedlines in FIG. 8 with respect to three-dimensional CAD data of theworkpiece model CWM in FIG. 7, and the centroid of the virtual cuboidIBX is set as the origin O of the workpiece model.

Fundamental Direction Image Selection Portion 8 e

The number of fundamental direction images is not necessarily six, andat least a plurality of fundamental direction images may be used. Forexample, in a case where opposing faces have the same shape like acuboid, only a fundamental direction image viewed from one face may beused. In other words, a fundamental direction image including the sameshape can be excluded, and thus a processing load in a three-dimensionalsearch can be reduced. Such a function of deleting a fundamentaldirection image of which a viewing way is the same as that of a certainfundamental direction image is realized by the fundamental directionimage selection portion 8 e. As an example, fundamental direction imagesacquired on the basis of the workpiece model CWM in FIG. 7 areillustrated in FIGS. 9A to 9D. In these figures, FIG. 9A illustrates aheight image in which the workpiece model CWM in FIG. 7 is viewed fromthe positive direction of the X axis, FIG. 9B illustrates a height imagein which the workpiece model CWM is viewed from the positive directionof the Y axis, FIG. 9C illustrates a height image in which the workpiecemodel CWM is viewed from the positive direction of the Z axis, and FIG.9D illustrates a height image in which the workpiece model CWM is viewedfrom the negative direction of the Z axis. Herein, as the height image,a height image is used in which height information is expressed as aluminance value such that a point is brightened as the point becomeshigher, and a point is darkened as the point becomes lower.

Here, matching/mismatching between viewing ways is checked by generatingsix height images viewed from top and bottom (positive and negativedirections of the Z axis), front and rear (positive and negativedirections of the Y axis), and left and right (positive and negativedirections of the X axis) of a workpiece, and checking matchingtherebetween. Herein, rotation matching is checked at a pitch angle of90°, and a face having the same viewing way as that of another face isexcluded from a search model registration target. In the workpiece modelCWM in FIG. 7, a height image viewed from the negative direction of theX axis and a height image viewed from the negative direction of the Yaxis respectively have the same viewing ways as those of the heightimages in FIGS. 9A and 9B, and are thus excluded from search modelgeneration targets. In the above-described way, search modelscorresponding to the number of faces having different viewing ways aregenerated. Fundamental direction images obtained by excludingunnecessary height images as mentioned above are displayed in thesix-drawing display region 3 a. The six-drawing display region is notlimited to the name thereof, all of six faces of a workpiece model arenot necessarily displayed, and display even in an aspect in which aheight image having the same viewing way is excluded as an unnecessaryimage and five or less faces are displayed is also referred to asdisplay in the six-drawing display region in the present specification.

Search Model Registration Screen 130

Such exclusion may be manually performed by a user, may be automaticallyperformed by the robot setting apparatus side, or may be performedthrough a combination thereof.

For example, on a search model registration screen 130, illustrated inFIG. 10, in which fundamental direction images BDI are registered assearch models for a three-dimensional search, the fundamental directionimage generation portion 8 e′ automatically generates the fundamentaldirection images BDI corresponding to six drawings, and displays thefundamental direction images in the six-drawing display region 3 a ofthe display unit 3. In this case, in a case where there is a commonfundamental direction image, a user is prompted to exclude thefundamental direction image from a search model registration target.Herein, a “registration target” checkbox 131 is provided for each of thefundamental direction images BDI which are search model candidates inthe search model selection portion 8 i which sets whether or not asearch model is to be registered. If a user checks the “registrationtarget” checkbox 131, the fundamental direction image BDI can beregistered as a search model, and if the user does not check the“registration target” checkbox 131, the fundamental direction image canbe excluded from a search model.

In this case, with respect to a fundamental direction image having thesame viewing way, the search model registration screen 130 is displayedto the user in a state in which the “registration target” checkbox 131is not checked in advance by the fundamental direction image selectionportion 8 e. In an initial state, the user may check the fundamentaldirection images BDI to be registered as search models and accurateselection of fundamental direction images to be excluded from searchmodels, and then may approve the selection or may perform correction orreplacement as necessary. As mentioned above, since a fundamentaldirection image to be registered as a search model for athree-dimensional search is selected, and a fundamental direction imagecandidate to be excluded from registration is presented, in a defaultmanner, labor-saving for search model registration work performed by auser can be achieved.

Actually Measured Data MWM

A description has been made of an example of using three-dimensional CADdata as a search model for a three-dimensional search. However, asdescribed above, the present invention is not limited tothree-dimensional CAD data as a search model, and, for example,three-dimensional data obtained by analyzing a plurality of pieces oftwo-dimensional CAD data or actually measured data obtained by actuallyimaging a workpiece in the sensor unit may be used as a search model. Asan example, FIG. 11 illustrates actually measured data MWM obtained byimaging a workpiece corresponding to the workpiece model CWM in FIG. 7from the positive direction of the X axis. As mentioned above, in a casewhere there is no CAD data of a workpiece, data obtained bythree-dimensionally measuring a real workpiece may be used. Asillustrated in FIG. 10, pieces of the actually measured data MWM of thenumber required for a three-dimensional search are registered throughimaging.

In a case where a real workpiece is registered, information (forexample, a shape of a floor of the workpiece vicinity) regarding abottom surface on which the workpiece is placed is three-dimensionallymeasured. Therefore, for example, preferably, unnecessary informationregarding the bottom surface is excluded by cutting out only a locationof a predetermined height or more from the bottom surface throughthreshold-value processing. Consequently, only a shape portion requiredfor a three-dimensional search can be registered.

Extraction of Feature Point

Next, a search model of a registered face is generated in a state inwhich a height image corresponding to each face of a search model targetworkpiece is registered as mentioned above. Herein, a feature pointrequired for a three-dimensional search is extracted for each registeredface. Herein, a description will be made of an example in which twotypes of feature points such as a feature point (a feature point on acontour) representing a contour of a shape and a feature point (afeature point on a surface) representing a surface shape are used as thefeature point. FIG. 12A illustrates a state in which two types offeature points are extracted with respect to a search model SM of aheight image (corresponding to FIG. 9A) viewed from the X axisdirection. FIG. 12B illustrates a state in which the search model SM isdisplayed in a perspective view. Here, feature points SCP on the surfaceand feature points OCP on the contour are preferably displayed indifferent display aspects. For example, the feature points SCP on thesurface are displayed white, and the feature points OCP on the contourare displayed light blue. Alternatively, other colors may be used, and,for example, the feature points on the contour may be displayed violetso as to be further differentiated from the feature points on thesurface displayed white. Since the feature points are displayed indifferent colors as mentioned above, a user can easily visuallydifferentiate implications of the respective feature points from eachother.

The feature points SCP on the surface are extracted from a surface of aworkpiece model at a predetermined interval. On the other hand,regarding the feature points OCP on the contour, for example, an edge ofa location or the like of which a height changes is extracted, and alocation further having undergone a thinning process is extracted as afeature point at a predetermined interval. As mentioned above, eachfeature point indicates a three-dimensional shape of a face.

Three-Dimensional Search Method

A three-dimensional search is performed by using the search model as aresult of extracting the feature points. Here, a description will bemade of a method of performing a three-dimensional search for extractingeach workpiece in a state in which three-dimensional shapes are acquiredby imaging a workpiece group in which a plurality of workpieces areloaded in bulk as illustrated in FIG. 13A or 13B as an input image.First, positions and attitudes (X, Y, Z, R_(X), R_(Y), and R_(Z)) in astate of each feature point of the search model most matching aresearched from the input image. Here, R_(X), R_(Y), and R_(Z)respectively indicate a rotation angle about the X axis, a rotationangle about the Y axis, and a rotation angle about the Z axis. Variousexpression methods for rotation angles are provided, but, herein, aZ-Y-X system Euler's angle is used (details thereof will be describedlater). The number of each of matching positions and attitudes is notnecessarily one for each search model, and a plurality of matchingpositions and attitudes of a predetermined extent or higher may bedetected.

Herein, a three-dimensional search is performed on as a workpiece groupwhich is displayed in a two-dimensional manner as in FIG. 13A or aworkpiece group which is displayed in a three-dimensional manner as inFIG. 13B in the input image, and the entire input image is searched.Thus, a searched image in which the workpiece group is displayed in atwo-dimensional manner as in FIG. 13C or displayed in athree-dimensional manner as in FIG. 13D is obtained as athree-dimensional search result. As illustrated in FIGS. 13C and 13D, itcan be seen that the feature points of the search model are searched forfrom the input image, and a workpiece corresponding to the search modelis detected. FIG. 13D illustrates a state in which search results of asearch model A and a search model B are obtained. The search model A andthe search model B correspond to search models A and B displayed in asearch model selection field in FIG. 14 which will be described later.For the right workpiece WK in FIG. 13D, two search results of the searchmodels A and B are obtained with the same workpiece. If grip is possibleat a grip position registered in each search model, a plurality of grippositions are obtained in this workpiece.

An image in which a workpiece is viewed for each face as in six drawingsis used as a search model used for a three-dimensional search, and thusit is possible to obtain an advantage that a calculation process of thethree-dimensional search can be simplified compared with a case of usinga perspective view or the like, and thus a process can be performed at ahigh speed with a reduced load. A state displayed in search modelregistration work is easily viewed, and can thus be easily visuallyrecognized by a user.

Evaluation Index of Three-Dimensional Search Result

An evaluation index of a three-dimensional search result may be set. Forexample, in the example illustrated in FIG. 13C or 13D, athree-dimensional search result is scored on the basis of to what degreecorresponding feature points are present in an input image (for example,on the basis of a proportion of the number of feature pointscorresponding to an error of a predetermined distance or less withrespect to a search result, or a value obtained by subtracting an erroramount of a feature point as a penalty according to a definedcomputation formula). In this method, a score lowers in a state in whichthere is a lot of invalid data (invalid pixel) which cannot bethree-dimensionally measured. As mentioned above, a score may be used asan index indicating the reliability of a three-dimensional searchresult. For example, workpieces are set to be preferentially gripped inan order of higher scores. There is a high probability that athree-dimensional search result of a predetermined score or less isdetermined as being wrongly detected, and thus a workpiece may be set tobe excluded from a grip target. For example, in the robot settingapparatus 100 in FIG. 6, an evaluation index calculation portion 8 q isprovided in the calculation unit 10 so as to calculate an evaluationindex for a search result on the basis of a predetermined criterion.Consequently, priority may be set in order from a search result with ahigh evaluation index, and a workpiece may be set to be grippedaccording to the priority. For example, a workpiece having the highestposition in the Z direction may be set to be preferentially grippedamong results of a predetermined score or higher. As a workpiece islocated at a higher position in the Z direction, the workpiece hardlyinterferes with other workpieces. Thus, a workpiece located at a highposition in the Z direction is set to be preferentially gripped, andthus it is possible to obtain an advantage that a processing load ofinterference determination can be reduced.

In the above-described way, each workpiece is detected from a workpiecegroup loaded in bulk, and thus a grip target workpiece can be recognizedby the robot setting apparatus side. Next, in order to grip a workpiecewith an end effector, it is necessary to recognize a grip position and agrip attitude of each workpiece. Thus, a grip position of a workpiece isregistered. In the present specification, registration of a “gripposition” and a “grip attitude” indicates registration of a positionwhere a workpiece is gripped and an attitude at that time. As a gripposition of a workpiece, one or more locations may be registered for theworkpiece. Registration of a grip position is preferably performed inthe face unit of a workpiece in terms of easiness of grip registrationwork or recognition of a grip position. In other words, an attitude of aworkpiece is defined to be an attitude with a specific face as asurface, and then a grip position is registered.

Grip Registration Screen 140

Here, FIGS. 14 and 15 illustrate examples of a user interface screens onwhich grip registration of registering a grip position at which an endeffector model grips a workpiece model is performed. A face of aworkpiece model to be registered is designated, and a grip attitude isregistered for each face, on a grip registration screen 140. The exampleillustrated in FIG. 14 is an example of a user interface screen on whichfour types of faces (respective faces of a search model) are designated,and a grip attitude is registered. Herein, a search model “C” isselected from among search models A to D, and three grip attitudes aredisplayed for the search model C.

In the grip registration screen 140 in FIG. 14, an image display field141 for displaying an image is provided on the left part, and anoperation field 142 for performing various operations is provided on theright part. The workpiece model CWM and an end effector model EEM aredisplayed in the image display field 141. A viewpoint can be changed bydragging the screen of the image display field 141. As mentioned above,the positioning portion 8 c has a function of adjusting a position or anattitude of a workpiece model displayed in a display region on a virtualthree-dimensional space. A display aspect in the image display field 141may be switched to two-dimensional display or three-dimensional display.In order to easily recognize the current display aspect, athree-dimensional reference coordinate axis BAX is displayed in theimage display field 141 in an overlapping manner. In the exampleillustrated in FIG. 14, a grip attitude registered in a grip attitude001 selected in the operation field 142, that is, a state in which apart of the workpiece model CWM is gripped by the end effector model EEMis displayed in the image display field 141. In this example, the entireend effector model EEM is displayed, but the entire end effector modelis not necessarily displayed during grip registration, and at least apart gripping the workpiece model CWM, for example, claws may bedisplayed.

The operation field 142 is provided with a search model selection field143 for selecting a search model, and a grip attitude display field 144displaying a grip attitude registered for a search model selected in thesearch model selection field 143. If a grip attitude is selected fromamong grip attitudes displayed in a list form in the search modelselection field 143, a registered grip attitude corresponding thereto isdisplayed in the image display field 141. If an editing button 145 ispressed, the registered grip attitude can be corrected. The operationfield 142 is further provided with an add button 146 for adding a gripattitude or a delete button 147 for deleting a registered grip attitude.If the delete button 147 is pressed, a selected grip attitude isdeleted.

Grip Attitude Adding Screen 150

In a case where a new grip attitude is desired to be added, the addbutton 146 is pressed. Consequently, as illustrated in FIG. 15, a gripattitude adding screen 150 is displayed, and thus a grip attitude can beadded. The end effector model EEM and the workpiece model CWM aredisplayed in the image display field 141. Grip attitude coordinateinformation 153 defining a grip attitude is displayed in the operationfield 142. Here, position parameters X, Y, Z, R_(X), R_(Y) and R_(Z)displayed as the grip attitude coordinate information 153 indicate dataof a position and an attitude of the end effector model EEM for theorigin of the search model. The origin of the search model may be thecentroid of the workpiece model CWM or a center coordinate of CAD dataas described above.

If values of X, Y, Z, R_(X), R_(Y) and R_(Z) of the grip attitudecoordinate information 153 are changed, the position and the attitude ofthe end effector model EEM which is three-dimensionally displayed in theimage display field 141 are updated according thereto. Conversely, ifthe end effector model EEM in the image display field 141 is dragged andmoved, the display content of the grip attitude coordinate information153 in the operation field 142 is updated to a grip attitude coordinateafter the movement. Consequently, a user can register a grip positionand a grip attitude while checking the end effector model EEM in theimage display field 141 and the grip attitude coordinate information 153in the operation field 142. The three-dimensional reference coordinateaxis BAX may be displayed in an overlapping manner in the image displayfield 141.

During registration of a grip position, a grip reference pointcorresponding to a position where the workpiece model CWM is gripped anda grip direction in which the workpiece model CWM is gripped by the endeffector model EEM are defined for the end effector model EEM displayedin the image display field 141. The grip reference point and the gripdirection are preferably set as default values on the robot settingapparatus 100 side. A position and an attitude of the end effector modelEEM are automatically adjusted according to a position and an attitudeof the workpiece model CWM on the grip attitude adding screen 150 inFIG. 15 such that the grip direction is orthogonal to a workpiece planerepresenting an attitude of the workpiece model CWM. This adjustment maybe performed by, for example, the three-dimensional pick determinationportion 8 l. Alternatively, a user may manually adjust a position and anattitude of the end effector model EEM by using the positioning portion8 c such that the grip direction of the end effector model EEM isorthogonal to the workpiece plane of the workpiece model CWM.

During registration of a grip position, in an initial state in which thegrip attitude adding screen 150 in FIG. 15 is open, a position and anattitude of the end effector model EEM have initial values in a state inwhich the end effector model EEM is directed downward, and the workpiecemodel CWM is located under the end effector model EEM. In theabove-described way, since a grip attitude is defined if the endeffector model EEM is moved down to be brought into contact with theworkpiece model CWM, an operation of a user side can be intuitivelyperformed, and thus it is possible to reduce the problem thatthree-dimensional alignment is troublesome. In other words, basically,only alignment of the end effector model EEM in the X, Y and Zdirections and rotation about the Z axis are set, and thus a gripattitude can be easily registered.

There may be a configuration in which a grip position at which the endeffector model EEM grips the workpiece model CWM is set, and thus aposition of the end effector model EEM is automatically adjusted suchthat the grip position is located on an axis extending from a gripdirection.

Registration of a grip attitude is performed with an image of aworkpiece as a reference. Thus, it is not necessary to use CAD data,and, as described above, a grip attitude can be registered for actuallymeasured data obtained by actually imaging a workpiece.

Fitting Function

When a grip position is designated, a user may manually perform work ofmoving an end effector model to a grip position of a workpiece model,and a fitting function of automatically performing the work may beprovided. In the related art, when grip positions X, Y and Z or gripattitudes R_(X), R_(Y) and R_(Z) at which an end effector model grips aworkpiece model are designated, for example, in the image display field141 in FIG. 15, the end effector model is dragged and moved until theend effector model is brought into contact with the workpiece model, ora numerical value is input to Z of the height direction in the operationfield 142, and thus setting is performed. However, work of a user movingan end effector model and fitting the end effector model to an attitudeat which a workpiece model is gripped with the naked eyes istroublesome, and, regarding designation of numerical values of positionparameters, it is hard to understand how the six position parameters arepreferably adjusted. Therefore, the fitting function of automaticallyperforming work of locating an end effector model to a grip position ofthe workpiece model is provided.

Here, in a case where a workpiece model or an end effector model hasheight information as in three-dimensional CAD data, a position on theworkpiece model desired to be gripped is designated through clicking ona mouse, a height direction of the position, that is, a Z coordinate isacquired, and a position obtained by adding an offset to the Zcoordinate is set as a Z coordinate after the end effector model ismoved. Consequently, time and effort for a user to manually move an endeffector model to a grip position or to manually input a positionparameter such as a Z coordinate can be reduced, and a grip position canbe accurately designated. As an example of the fitting function, a “fit”button 154 which is one type of relative position setting portion 8 d 5is disposed in the operation field 142 displayed on the grip attitudeadding screen 150 in FIG. 15. If the “fit” button 154 is pressed, theend effector model EEM is virtually moved in the Z direction, and isstopped at a position of being brought into contact with the workpiecemodel CWM, and a position obtained by subtracting an offset amount fromthe interference position in a reverse direction is set as a gripposition. The offset amount is a numerical value for slight separationin order to prevent the tip of the end effector model EEM frominterfering with the workpiece model CWM or being damaged due tocontact, and may be defined according to a workpiece or an application,such as 1 mm.

In the above-described example, a description will be made of the methodof performing grip registration by using height images used to generatea three-dimensional search model for performing a three-dimensionalsearch. Since a search model for extracting a workpiece and a model forregistering a grip position are commonized, a user can perform athree-dimensional search or setting of a grip position for a commonworkpiece, and can thus obtain a unified operation feeling, and canintuitively easily understand the common workpiece. However, in thepresent invention, a search model for a three-dimensional search doesnot necessarily match a grip registration target model. If acorrespondence relationship between a three-dimensional search model anda model used for grip registration is known, a model used as a searchmodel is not necessarily used for grip registration.

Checking of matching between viewing ways after a height image isgenerated on the basis of three-dimensional CAD data is not limited tochecking using positive and negative directions of the respective axes.For example, a workpiece having, for example, a cuboid shape has thesame viewing way even when viewed from any direction such as positiveand negative directions of the X axis, positive and negative directionsof the Y axis, and positive and negative directions of the Z axis, andthus a search model for a single face may be generated. Similarly, alsoregarding registration of a grip position, grip may be designated for asingle face. Therefore, all faces (the positive and negative directionsof the X axis, the positive and negative directions of the Y axis, andthe positive and negative directions of the Z axis) are in a state ofbeing able to be gripped. For example, with respect to workpieces WKCeach having a cuboid shape as illustrated in FIG. 16A and loaded inbulk, one face of the cuboid is registered as a model as illustrated inFIG. 16B, and a grip attitude of gripping the face directly from the tophas only to be registered. Thus, as illustrated in FIG. 16C, it ispossible to perform a three-dimensional search on all of the six facesand for the end effector model EEM to grip all of the six faces.

Calibration

In the above-described grip registration screen 140, relative positionand attitude of the end effector model EEM for a workpiece duringgripping are registered with respect to the origin of a search model. Onthe other hand, when a workpiece is picked by a real end effector, avision coordinate which is a coordinate of a three-dimensional space(vision space) in which the workpiece is imaged by the sensor unit isrequired to be converted into a robot coordinate used for the robotcontroller 6 to actually drive the robot. Specifically, a position andan attitude of a workpiece obtained as a result of a three-dimensionalsearch are obtained as a position (X,Y,Z) and an attitude(R_(X),R_(Y),R_(Z)) in the vision space (the attitude(R_(X),R_(Y),R_(Z)) indicates an attitude expressed by a Z-Y-X systemEuler's angle which will be described later). An attitude of the endeffector gripping the workpiece is also obtained as a position (X,Y,Z)and an attitude (R_(X),R_(Y),R_(Z)) in a virtual three-dimensional spaceof the robot setting apparatus. In order for the robot controller 6 todrive the robot, the position and the attitude in the vision space arerequired to be converted into a position (X′,Y′,Z′) and an attitude(R_(X)′,R_(Y)′,R_(Z)′) in the robot space. A process of calculating aconversion formula for converting a position and an attitude calculatedin the coordinate system displayed by the robot setting apparatus into aposition and an attitude in the coordinate system used for the robotcontroller to operate an end effector is referred to as calibration.

Embodiment 2

FIG. 17 is a functional block diagram illustrating an example of a robotsystem having a calibration function between a robot setting apparatus(machine vision apparatus) and a robot as Embodiment 2. In FIG. 17, arobot system 2000 includes a robot setting apparatus 200, a display unit3, an operation unit 4, a sensor unit 2, a robot controller 6, and arobot RBT. In the robot system illustrated in FIG. 17, a member commonto FIG. 6 or the like described above is given the same referencenumeral, and a detailed description thereof will be omitted asappropriate.

Robot Setting Apparatus 200

The robot setting apparatus 200 includes an input image acquisition unit2 c, a storage unit 9, a calculation unit 10, an input/output interface4 b, a display interface 3 f, and a robot interface 6 b.

The input image acquisition unit 2 c acquires an input image including athree-dimensional shape on the basis of an image including an endeffector measured by the sensor unit. When the input image acquisitionunit 2 c acquires an input image of the end effector, the input imagepreferably includes an attachment position attached to a flange surfaceat a tip of an arm portion of the robot. The input image is captured atan attitude at which an area of the end effector is increased. Forexample, a user operates the robot such that the input image of the endeffector is captured at an attitude of being sideways as a horizontalattitude.

The calculation unit 10 includes an end effector model registrationportion 8 u, a workpiece model registration portion 8 t, a calibrationportion 8 w, a search model registration portion 8 g, athree-dimensional search portion 8 k, a conversion portion 8 x, an endeffector attachment position correction portion 8 y, and a grip positionspecifying portion 8 d.

The end effector model registration portion 8 u is a member forregistering an end effector model which is three-dimensional CAD dataand virtually expresses a three-dimensional shape of an end effector.For example, the end effector model registration portion 8 u readsthree-dimensional CAD data indicating a shape of an end effector createdseparately, and registers the three-dimensional CAD data as an endeffector model. In this case, the end effector model registrationportion 8 u registers three-dimensional CAD data which is input via theinput/output interface 4 b as an end effector model. Alternatively,three-dimensional point group data obtained by imaging a real endeffector may be used as an end effector model. In this case, the endeffector model registration portion 8 u registers three-dimensionalpoint group data acquired in the sensor unit 2 or the input imageacquisition unit 2 c as an end effector model. Alternatively,three-dimensional CAD data simulating an end effector may be created andregistered. In this case, the end effector model registration portion 8u realizes a function of a simple three-dimensional CAD.

The calibration portion 8 w is a member for acquiring calibrationinformation for converting a position and an attitude calculated in acoordinate system of a vision space which is a virtual three-dimensionalspace displayed on the display unit into a position and an attitude in acoordinate system of a robot space in which the robot controlleroperates an end effector.

The calibration portion 8 w calculates a conversion formula between anactual position coordinate of the end effector EET of the robot and aposition coordinate on an image displayed on the robot setting apparatuswith respect to a plurality of position coordinates. A coordinateconversion method is not particularly limited, and a known method suchas three-dimensional affine conversion may be used as appropriate.

The storage unit 9 stores calibration information from the calibrationportion 8 w.

The three-dimensional search portion 8 k is a member for performing athree-dimensional search for specifying an image region corresponding toa position and an attitude of an end effector from an input imageacquired by the input image acquisition unit 2 c by using an endeffector model as a search model.

The conversion portion 8 x converts a vision coordinate into a robotcoordinate on the basis of calibration information obtained by thecalibration portion 8 w. The conversion portion 8 x reads thecalibration information stored in the storage unit 9.

The end effector attachment position correction portion 8 y correctserrors between a position and an attitude on the vision space of the endeffector and a position and an attitude on the robot space by usinginformation obtained by converting the position and the attitude of theend effector on the vision space searched for by the three-dimensionalsearch portion into the position and the attitude on the robot space inthe conversion portion 8 x. Consequently, since a three-dimensionalsearch is performed by using an end effector model, an error of avirtual end effector model can be corrected by taking into considerationan attachment state of a real end effector, and thus more accuratesetting work can be performed.

The grip position specifying portion 8 d specifies one or more grippositions at which a workpiece model registered by the workpiece modelregistration portion 8 t is gripped by an end effector model.

Z-Y-X System Euler's Angle

Here, a Z-Y-X system Euler's angle will be described. In the relatedart, in order to define a grip position at which a workpiece is grippedby an end effector, the Z-Y-X system Euler's angle is used forpositioning of an attitude of a workpiece model or an end effector modelon the basis of three-dimensional CAD data. In this case, a position andan attitude of the end effector model for the workpiece model areexpressed by six position parameters (X, Y, Z, R_(X), R_(Y), and R_(Z)).Here, X, Y, and Z indicate orthogonal coordinate axes defining athree-dimensional space, and R_(X), R_(Y), and R_(Z) respectivelyindicate rotation angles obtained through rotation centering on the Xaxis, the Y axis, and the Z axis.

Here, with reference to FIGS. 18 to 21, a description will be made of anexample of rotating an end effector model according to the Z-Y-X systemEuler's angle. First, FIG. 18 illustrates a state in which the endeffector model EEM and a workpiece model WM11 are displayed in an imagedisplay region of the display unit 3. In this example, for betterunderstanding of description, reference coordinate axes BAX (XYZ)defining a virtual three-dimensional space and rotation-completedcoordinate axes RAX (XYZ) which are three-dimensional space coordinateaxes after a rotation target (here, the end effector model EEM) isrotated are separately displayed.

In the example illustrated in FIG. 18, since the workpiece model WM11side is stopped, and the end effector model EEM side is rotated, thereference coordinate axes BAX express a position and an attitude of theworkpiece model WM11, and the rotation-completed coordinate axes RAXexpress a position and an attitude of the end effector model EEM. In therotation-completed coordinate axes RAX of the end effector model EEMside, the origin thereof matches a grip reference point HBP which is agrip position of the end effector model EEM (details thereof will bedescribed later).

In a state before rotation, the rotation angles R_(X), R_(Y), and R_(Z)about the XYZ axes are R_(X)=0°, R_(Y)=0°, and R_(Z)=0°, and thereference coordinate axes BAX match the rotation-completed coordinateaxes RAX. If the end effector model EEM is rotated counterclockwise by90°centering on the Z axis in this state, a rotation result is asillustrated FIG. 19. In a state illustrated in FIG. 19, the rotationangles R_(X), R_(Y) and R_(Z) are R_(X)=0°, R_(Y)=0°, and R_(Z)=90°, andthus the rotation-completed coordinate axes RAX displayed in the imagedisplay region are updated to an attitude after the rotation. If the endeffector model EEM is rotated clockwise by 90°centering on the Y axis inthe state illustrated in FIG. 19, a rotation result is as illustrated inFIG. 20. In this state, the rotation angles R_(X), R_(Y) and R_(Z) areR_(X)=0°, R_(Y)=90°, and R_(Z)=90°, and thus the rotation-completedcoordinate axes RAX are also updated to an attitude after the rotation.If the end effector model EEM is rotated clockwise by 90°centering onthe X axis in the state illustrated in FIG. 20, a rotation result is asillustrated in FIG. 21, and the rotation angles R_(X), R_(Y) and R_(Z)are R_(X)=90°, R_(Y)=90°, and R_(Z)=90.

In the related art, a workpiece model or an end effector model isgenerated or a position thereof is expressed by using the Z-Y-X systemEuler's angle. However, the number of operable position parameters issix and is large, and thus work of a user adjusting a position or anattitude to an expected position or an expected attitude by adjustingthe position parameters is not easy. The XYZ axes used to move aposition while an attitude is maintained are different from the R_(X),R_(Y) and R_(Z) axes used for rotation, and thus there is a problem inthat, if the position parameters are changed, it is hard to intuitivelyrecognize a change result. This state will be described with referenceto FIG. 22.

Now, in a state illustrated in FIG. 21, the rotation-completedcoordinate axes RAX which are orthogonal coordinate axes after rotationby the rotation angles R_(X)=90°, R_(Y)=90°, and R_(Z)=90° are displayedin the image display region. If R_(Y) or R_(Z) is changed in this state,rotation is performed centering on axes which are different from thedisplayed rotation-completed coordinate axes RAX. For example, R_(Y)rotation is performed centering on the Y axis (the Y axis of therotation-completed coordinate axes RAX in FIG. 19) before R_(X) rotationas indicated by a dashed line in FIG. 22, instead of the Y axis of therotation-completed coordinate axes RAX in FIG. 22. R_(Z) rotation is notperformed centering on the Z axis of the rotation-completed coordinateaxes RAX in FIG. 22, and is performed centering on the Z axis (the Zaxis of the rotation-completed coordinate axes RAX in FIG. 18) beforeR_(X) or R_(Y) rotation indicated by dashed lines.

This is because the Z-Y-X system Euler's angle is defined on the basisof rotation being performed in an order of the Z axis, the Y axis, andthe X axis. According to the Z-Y-X system Euler's angle, the Z axis isused as a reference, and rotational axes of R_(X) and R_(Y) are rotateddue to R_(Z) rotation about the Z axis. Here, a rotational axis aboutthe Z axis is not moved, but rotational axes of R_(X) and R_(Y) arerotated due to R_(Z) rotation about the Z axis. If rotation is performedby R_(Y) about the Y axis determined according thereto, the X axis isthus also rotated. Here, since a rotational axis about the Y axis isdefined depending on R_(Z) rotation about the Z axis, even if rotationis performed by R_(X) about the X axis, a rotational axis about the Yaxis and a rotational axis about the Z axis are not changed. In otherwords, in the Z-Y-X system Euler's angle, it can be said that R_(Z) hasan independent rotational axis, R_(Y) has a rotational axis whichdepends on R_(Z), and R_(X) has a rotational axis which depends onR_(Y).

As mentioned above, in the related art, in the Z-Y-X system Euler'sangle generally used for control of a robot, since the three rotationalaxes are correlated with each other, and an axis after rotation aboutanother rotational axis is also rotated, a user hardly recognizes whichaxis is a rotational axis, and thus it is not easy to perform rotationas intended.

Display of Correction Rotational Axis

In contrast, in the method according to the present embodiment,rotational axes of R_(X), R_(Y), and R_(Z) are displayed with actualrotational axes as references. Unlike the axes being displayed in athree-dimensional space using the Z-Y-X system Euler's angle of therelated art (FIGS. 18 to 21), a real rotational axis which is correctedby taking into consideration a state after rotation about anotherrotational axis is displayed as a correction rotational axis. Forexample, a description will be made of an example of displaying acorrection rotational axis of R_(Z) in a state illustrated in FIG. 22.In this case, as illustrated in FIG. 23, the Z axis of when R_(X) andR_(Y) are computed to be 0° is a correction rotational axis. Acorrection rotational Z axis A×Z displayed here is the Z axis of whenR_(X) is 0°, R_(Y) is 0°, and R_(Z) is 90°.

On the other hand, in a case where a correction rotational axis of R_(Y)is displayed in the state illustrated in FIG. 22, the Y axis of whenR_(X) is computed to be 0° is a real rotational axis as illustrated inFIG. 24. A correction rotational Y axis A×Y displayed here is the Y axisof when R_(X) is 0°, R_(Y) is 90°, and R_(Z) is 90.

On the other hand, in a case where a correction rotational axis of R_(X)is displayed in the state illustrated in FIG. 22, the X axis may bedisplayed without being changed. FIG. 25 illustrates an example in whicha correction rotational X axis as a correction rotational axis isdisplayed. A correction rotational X axis AXX displayed here is the Xaxis of when R_(X) is 90°, R_(Y) is 90°, and R_(Z) is 90°.

A start point of the correction rotational axis is preferably the centerof rotation of an end effector model. For example, an intermediateposition of a pair of claws which is a grip position of the end effectormodel EEM is used as the start point.

X-Y-Z Euler's Angle

Regarding the correction rotational axis, a rotational axis may bedisplayed in the same manner even in a case where a rotation order ischanged. For example, in the examples illustrated in FIGS. 23 to 25, theZ-Y-X system Euler's angle is used, and thus rotation is performed in anorder of R_(Z), R_(Y), and R_(X). On the other hand, since if an X-Y-ZEuler's angle is used, rotation is performed in an order of R_(X),R_(Y), and R_(Z), an axis computed to be 0° is different from that inthe Z-Y-X system Euler's angle. For example, assuming that rotation isperformed in the order of R_(X), R_(Y), and R_(Z) in the same manner asin FIGS. 18 to 21 in order to obtain correction rotational axes with theX-Y-Z Euler's angle, in a case where the Z axis is displayed as acorrection rotational axis, the Z axis after rotation is displayedwithout being changed. In a case where the Y axis is displayed as acorrection rotational axis, the Y axis of when R_(Z) is computed to be0° is displayed. In a case where the X axis is displayed as a correctionrotational axis, the X axis of when R_(Z) and R_(Y) are computed to be0° is displayed.

When a user sets a grip position or a grip attitude by using an Euler'sangle in the above-described way, a rotational axis can be displayed tobe easily understandable, and thus it is possible to perform anintuitive operation without understanding the complex concept of anEuler's angle.

Embodiment 3

It is not easy for a user to set a position or an attitude of an endeffector or a workpiece defined by a plurality of position parameters,as in an Euler's angle. Therefore, setting procedures of positionparameters may be guided such that a user can designate a restrictedposition parameter among the plurality of position parameters, and anecessary position parameter may be sequentially set according to theguide. Such an example is illustrated in FIG. 26 as a robot system 3000according to Embodiment 3. The robot system 3000 illustrated in FIG. 26includes a robot setting apparatus 300, a display unit 3B, an operationunit 4, a sensor unit 2, a robot controller 6, and a robot RBT. In therobot system illustrated in FIG. 26, a member common to FIG. 6 or thelike described above is given the same reference numeral, and a detaileddescription thereof will be omitted as appropriate.

Robot Setting Apparatus 300

The robot setting apparatus 300 includes an input image acquisition unit2 c, a calculation unit 10, a storage unit 9, an input/output interface4 b, a display interface 3 f, and a robot interface 6 b. The storageunit 9 includes a grip position storage portion 9 b. The grip positionstorage portion 9 b is a member for storing a grip position of aworkpiece model or an end effector model designated by a grip positionspecifying portion 8 d.

The calculation unit 10 includes the grip position specifying portion 8d, a grip position copying portion 8 d 8, a relative position settingportion 8 d 5, a search model registration portion 8 g, athree-dimensional search portion 8 k, a three-dimensional pickdetermination portion 8 l, and a section model generation portion 8 s.

The grip position specifying portion 8 d is a member for designating sixposition parameters including an X coordinate, a Y coordinate, and a Zcoordinate which are respectively coordinate positions on the X axis,the Y axis, and the Z axis, and an R_(X) rotation angle, an R_(Y)rotation angle, and an R_(Z) rotation angle centering on the X axis, theY axis, and the Z axis, for specifying a position and an attitude of oneor both of a workpiece model or an end effector model displayed on avirtual three-dimensional space on the display unit.

The grip position copying portion 8 d 8 is a member for reading a gripposition of a workpiece model or an end effector model stored in thegrip position storage portion 9 b, changing the grip position, andregistering the changed grip position as a new grip position.Consequently, when a plurality of grip positions are registered, a gripposition which is already registered is read, and a grip position ischanged on the basis of the grip position so as to be registered as anew grip position. Therefore, a grip position can be more easily addedthan a case where a grip position is registered from the beginning, andthus it is possible to achieve labor-saving of registration work.

The three-dimensional pick determination portion 8 l is a member fordetermining whether or not an end effector can grip a workpiece model ata grip position designated for the workpiece model by the grip positionspecifying portion 8 d with respect to each search result obtainedthrough a search in the three-dimensional search portion 8 k.

The section model generation portion 8 s generates a section modelformed of sets of a plurality of sections and section positions, each ofthe sections being obtained by cutting an end effector model in a planeorthogonal to a fundamental axis along the fundamental axis which is setto be linear in one direction for the end effector model, and each ofthe section positions being a point at which an orthogonal planeincluding each section intersects the fundamental axis. Polygon data orthe like is used for an end effector model for creating the sectionmodel.

Grip Position Specifying Portion 8 d

The grip position specifying portion 8 d illustrated in FIG. 26 includesa first designation portion 8 d 9, a second designation portion 8 d 6,and a third designation portion 8 d 7. The first designation portion 8 d9 is a member which can designate at least one of an X coordinate, a Ycoordinate, and a Z coordinate, and cannot designate other positionparameters. The second designation portion 8 d 6 is a member which candesignate at least one of an R_(X) rotation angle, an R_(Y) rotationangle, and an R_(Z) rotation angle, and cannot designate other positionparameters, with respect to a workpiece model or an end effector modelfor which some position parameters are designated by the firstdesignation portion 8 d 9. The third designation portion 8 d 7 is amember which can designate a position parameter not designated by thesecond designation portion 8 d 6 among at least some of the R_(X)rotation angle, the R_(Y) rotation angle, and the R_(Z) rotation angle,and cannot designate other position parameters, with respect to aworkpiece model or an end effector model for which some positionparameters are designated by the first designation portion 8 d 9 and thesecond designation portion 8 d 6.

Consequently, regarding designation of a grip position, all of the sixposition parameters are not designated on a single screen, positionparameters which can be designated are designated separately by aplurality of restricted designation portions, and thus it is possible toprevent a situation in which a plurality of position parameters areintertwined with each other, so that a position or an attitude is hardlyrecognized, and position parameters are sequentially designated suchthat information necessary for specifying a position and an attitude canbe set. Particularly, since any part on a planar workpiece modeldisplayed on the display unit is designated by the first designationportion 8 d 9 through mouse clicking or the like as an initial positionof a grip position, a user can perform designation in a visually easilyunderstandable method such as direct designation of a workpiece modelwith an attitude at which the workpiece model is easy to view, and thuswork can be considerably simplified compared with complex and troublework such as fixation of an attitude or definition of a numerical valueas in the related art. In the first designation portion 8 d 9, aworkpiece model is not limited to an aspect of being displayed in atwo-dimensional form, and may be displayed in a three-dimensional form.

Procedures in Setting Work

Here, in relation to setting work for operating the robot system,procedures of teaching work performed before an actual operation will bedescribed with reference to a flowchart of FIG. 27.

First, in step S2701, a search model for three-dimensionally searchingfor a workpiece is registered. Here, as the search model, a workpiecemodel as described above, for example, three-dimensional CAD data may beregistered. Alternatively, actually measured data obtained by actuallyimaging a workpiece in the sensor unit may be registered as the searchmodel.

Next, in step S2702, an end effector model of the robot is registered.Herein, three-dimensional CAD data may be registered as the end effectormodel. Next, in step S2703, a face of the workpiece model to be grippedis selected from a height image. Next, in step S2704, a position and anattitude of the robot of when the selected face is gripped areregistered. Next, in step S2705, it is determined whether or notpositions and attitudes of the necessary number are registered, and, ina case where positions and attitudes of the necessary number are notregistered, the flow returns to step S2703, and the process isrepeatedly performed. In a case where positions and attitudes of thenecessary number are registered, the process is finished.

Procedures of Registering Three-Dimensional CAD Data as Search Model

Here, in step S2701, a description will be made of examples ofprocedures of registering a search model of a workpiece in a case wherethree-dimensional CAD data is used as a search model with reference to aflowchart of FIG. 28. First, in step S2801, a three-dimensional CAD datamodel of the workpiece is read. Next, in step S2802, the center of acircumscribing cuboid of the three-dimensional CAD data model iscorrected to the origin of the three-dimensional CAD data. In stepS2803, height images viewed from respective directions of “top”,“bottom”, “left”, “right”, “front”, and “rear” are generated. Here, in acase where a height image is generated on the basis of thethree-dimensional CAD data, the height image is generated such that theorigin of CAD is the center of the height image.

Next, in step S2804, a height image having the same viewing way isdeleted from the generated height images. Finally, in step S2805, asearch model is registered by using the generated height images.

In the above-described way, a user can register a search model used fora three-dimensional search according to a guidance.

Registration of End Effector Model

Next, a description will be made of details of procedures of registeringan end effector model in step S2702 in FIG. 27 with reference to aflowchart of FIG. 29. Herein, three-dimensional CAD data forming an endeffector model is formed of polygon data.

First, in step S2901, polygon data of the end effector model is read.Next, in step S2902, a direction in which a section is to be created isdetermined. In step S2903, a section model is created. The section modelgeneration portion 8 s in FIG. 26 creates the section model. Details ofcreation of the section model will be described later. In theabove-described way, the end effector model is registered in the storageunit 9.

Additional Region

When an end effector model is registered, an additional region may beadded to an original end effector model. Procedures thereof will bedescribed with reference to a flowchart of FIG. 30. First, in stepS3001, the end effector model is registered. For example, an endeffector model formed of three-dimensional CAD data such as STL data isread.

Next, in step S3002, an additional region is set. The additional regionis used to add a shape of a real end effector or a shape simulating acover or a seat belt added thereto, to a surface of the end effectormodel, for example, during interference determination (which will bedescribed later in detail), and thus the accuracy of interferencedetermination can be improved.

Embodiment 4

An additional model creation function is used to set such an additionalregion. An example of a robot system having the additional modelcreation function is illustrated in a block diagram of FIG. 31 asEmbodiment 4. A robot system 4000 illustrated in FIG. 31 includes arobot setting apparatus 400, a display unit 3, an operation unit 4, asensor unit 2, a robot controller 6, and a robot RBT. In the robotsystem illustrated in FIG. 31, a member common to FIG. 6 or the likedescribed above is given the same reference numeral, and a detaileddescription thereof will be omitted as appropriate.

Robot Setting Apparatus 400

The robot setting apparatus 400 includes an input image acquisition unit2 c, a calculation unit 10, a storage unit 9, an input/output interface4 b, a display interface 3 f, and a robot interface 6 b.

The calculation unit 10 includes a workpiece model registration portion8 t, an end effector model registration portion 8 u, an additional modelcreation portion 8 v, a grip position specifying portion 8 d, a searchmodel registration portion 8 g, a three-dimensional search portion 8 k,an interference determination portion 8 m, and a section modelgeneration portion 8 s.

The additional model creation portion 8 v is a member for creating anadditional model in which an additional region expressed by one or morepredefined solid figures is added to a surface of an end effector model.Consequently, an interference determination region with a simple shapesuch as a cuboid or a cylinder is added to an end effector model withoutediting three-dimensional CAD data, interference determinationcorresponding to each region is performed, so that interferencedetermination can be easily performed. The solid figures include notonly fundamental figures prepared on the robot setting apparatus side inadvance, but also figures which can be freely designed by a user.

The grip position specifying portion 8 d is a member for specifying oneor more grip positions at which a workpiece model is gripped by an endeffector model for the workpiece model registered by the workpiece modelregistration portion 8 t.

The three-dimensional search portion 8 k is a member for performing athree-dimensional search for specifying an image region corresponding toa position and an attitude of each workpiece by using a search modelregistered by the search model registration portion 8 g from an inputimage acquired by the input image acquisition unit 2 c.

The interference determination portion 8 m is a member for performinginterference determination for determining the presence or absence ofinterference with another object which may hinder an operation when anend effector is operated by using an additional model created by theadditional model creation portion 8 v. The interference determinationportion 8 m performs interference determination on an image including aperipheral object of a workpiece in an input image in order to determinewhether or not the peripheral object of the workpiece interferes with anend effector when the end effector is moved to a position in order togrip any one of workpieces included in the input image acquired by theinput image acquisition unit 2 c. Consequently, interferencedetermination can be performed through comparison between an endeffector model and an input image. The interference determinationportion 8 m determines, for example, the presence or absence ofinterference with an object present in the vicinity of a workpiece in acase where an end effector model is disposed at a grip positionspecified by the grip position specifying portion 8 d in order to gripthe workpiece with respect to a search result corresponding to aposition and an attitude of each workpiece searched for from an inputimage by the three-dimensional search portion 8 k.

Additional Region Setting Screen 310

An additional region is added to an end effector model formed ofthree-dimensional CAD data. Here, FIG. 32 illustrates an additionalregion setting screen 310 for adding an additional region to an endeffector model as an aspect of the additional model creation portion 8v. On the additional region setting screen 310, the end effector modelEEM which is three-dimensional CAD data of an end effector may be read,and a figure may be added thereto. The additional region setting screen310 includes an image display field 141 and an operation field 142. Thethree-dimensional CAD data of the read end effector model EEM isdisplayed in the image display field 141. The operation field 142 isprovided with a basic figure display field 311. A list of selectablebasic figures is displayed in the basic figure display field 311. Thebasic figures may include a cuboid, a cylinder, a circular cone, atriangular cone, and a hexagonal prism. A user can add a basic figure toa designated position on the end effector model EEM by selecting thedesired basic figure from the basic figure display field 311 andpressing an “add” button 312. An “edit” button 313 is provided in eachof rows displayed in a list form in the basic figure display field 311,and, if the button is pressed, setting of a selected basic figure can bechanged. For example, basic figure parameters such as a length of oneside of a bottom, a diameter, and a height can be adjusted.

In the example illustrated in FIG. 32, a cuboid is selected as a basicfigure desired to be added, and is highlighted in the basic figuredisplay field 311. If the “edit” button 313 is pressed in this state, abasic figure editing screen 320 in FIG. 33 is displayed. A basic figureparameter adjustment field 321 is provided in the operation field 142 ofthe basic figure editing screen 320, and a size or a basic figure or aposition or an attitude of an end effector for a reference position canbe changed therefrom. If a basic figure parameter is adjusted from thebasic figure parameter adjustment field 321, the display content of theimage display field 141 is also updated according thereto.

In the example illustrated in FIG. 32, an additional region ADA isinserted between a flange surface FLS at the tip of the arm portion andthe end effector model EEM. Consequently, a state in which an attachmenttool or the like is interposed between the arm portion and the endeffector can be reproduced. In a case where the flange surface at thetip of the arm portion is deviated from a surface set for calculation onthe robot setting apparatus side, a basic figure may be used to offset adeviation amount.

A plurality of figures may be added. In the example illustrated in FIG.32, various basic figures may be added by selecting a desired basicfigure from the basic figure display field 311 and pressing the “add”button 312. Consequently, various shapes can be expressed through acombination of a plurality of basic figures.

As mentioned above, since the function of setting an additional regionfor an end effector model is provided, a shape can be easily added byonly adding a basic figure prepared in advance without changing a shapethrough direct editing of a shape of three-dimensional CAD data as inthe related art. As a result, in interference determination of checkingwhether or not a workpiece interferes with a peripheral object inadvance when an end effector is disposed at a grip position of grippingthe workpiece, in a case where there is a member which is not includedin an end effector model but is actually present, the accuracy of adetermination result can be improved by using a shape simulating themember. For example, in a case where an end effector is connected to thetip of the arm portion via a joint, the entire end effector may beoffset, and thus the tip may protrude slightly. There is a case where anadditional element such as a cover for contact prevention or a cableextending from an end effector is present on the outside of the endeffector. Such an additional element is often not included inthree-dimensional CAD data of an end effector model, and CAD data is notoften prepared separately. Thus, it takes time to process and editthree-dimensional CAD data of an end effector model such that a shapecorresponding to the additional element is obtained. Even if a complexshape can be expressed by CAD data, if three-dimensional CAD dataindicating a complex shape is used for interference determination, acalculation process becomes complex, and a processing time is increased.

Therefore, if such an additional element is expressed as a figure, anend effector model having a form close to an actual state can beobtained with a simple additional region without performing atroublesome editing work, and thus a shape suitable for a real shape canbe easily expressed. According to the method, a form of CAD data is notchanged, and an interference determination region is added, and aprocessing time increase amount is reduced.

Grip Position Setting Procedures

Next, in step S2704 in FIG. 27, a description will be made of proceduresof registering a grip position and a grip attitude at which a workpieceis gripped by an end effector with reference to a flowchart of FIG. 34and FIGS. 35A to 38. Herein, six position parameters, that is, gripposition parameters X, Y and Z and grip attitude parameters R_(X), R_(Y)and R_(Z) defining a position and an attitude (hereinafter, collectivelyreferred to as a “grip position”) of the end effector model EEM for theworkpiece model WM11 of when the workpiece model WM11 is gripped in astate in which the workpiece model WM11 is fixed by using the workpiecemodel WM11 and the end effector model EEM formed of three-dimensionalCAD data, are sequentially set according to procedures illustrated inthe flowchart of FIG. 34.

X-Y Designation Unit

First, in step S3401, an X coordinate and a Y coordinate of a gripposition are designated. Herein, a two-dimensional plane image ontowhich a workpiece is projected onto a plane with an X-Y designation unitis displayed, and a grip position is designated on the plane image. AnX-Y designation screen 230 as an aspect of the X-Y designation unit isillustrated in FIG. 35B. On the X-Y designation screen 230 illustratedin FIG. 35A, three-dimensional CAD data of a workpiece is displayed in aplan view in the image display field 141. In this state, a userdesignates a part of a workpiece model WM11F desired to be gripped by anend effector model and displayed in the plan view. If a grip position isselected with a pointing device such as a mouse on the plan view as inFIG. 35B, a desired position can be designated in an easilyunderstandable aspect without being aware of an attitude, a rotationangle, or the like. Particularly, since a grip position on the workpiecemodel WM11F displayed in the plan view is clicked and designated, it ispossible to obtain an advantage that easy designation can be performedcompared with a case of adjusting a position by inputting a numericalvalue. A user may determine X and Y coordinates of a grip position onthe basis of a manually designated position and a scale of an image. AnX and Y coordinate display field on which X and Y coordinates aredisplayed in numerical values may be provided on the X-Y designationscreen.

Z-R_(Z) Designation Unit

Next, in step S3402, a grip position Z and a grip attitude R_(Z) aredesignated from a Z-R_(Z) designation unit. FIG. 36 illustrates aZ-R_(Z) designation screen 240 as an aspect of the Z-R_(Z) designationunit. The Z-R_(Z) designation screen 240 illustrated in FIG. 36 includesan image display field 141 and an operation field 142. The operationfield 142 is provided with a Z coordinate designation field 241 fordesignating a Z coordinate of a grip position and an R_(Z) rotationangle designation field 242 for designating an R_(Z) rotation angle of agrip attitude. In this stage, the X and Y coordinates are alreadydetermined in step S3401, and thus only a Z axis may be displayed. Thus,only the correction rotational Z axis AXZ as a correction rotationalaxis is displayed in an overlapping manner in a state of displaying theend effector model EEM and the workpiece model WM11 in the image displayfield. Consequently, a user can imagine how the end effector model EEMdisplayed in the image display field 141 is rotated when a rotationangle is changed, and thus it is possible to obtain an advantage thatposition adjustment work for the end effector model EEM can be easilyperformed. Particularly, the user can visually recognize a rotationdirection without being confused since other rotational axes are notdisplayed. Since position parameters other than the Z coordinate and theR_(Z) rotation angle cannot be designated, it is possible to preventother position parameters from being wrongly designated, and thus a usercan concentrate on and set only position parameters that can bedesignated, and can thus focus on only necessary setting in a state inwhich confusion such as misunderstanding and erroneous setting iseliminated.

Regarding rotation of the end effector model EEM, a numerical value isinput to the R_(Z) rotation angle designation field 242, and thus theend effector model EEM in the image display field 141 is automaticallyrotated and displayed. The end effector model EEM displayed in the imagedisplay field 141 is dragged and rotated, and thus a value of the R_(Z)rotation angle displayed in the R_(Z) rotation angle designation field242 is also changed.

R_(Y) Designation Unit

In step S3403, a grip attitude R_(Y) is designated. Herein, an R_(Y)rotation angle is designated from an R_(Y) rotation angle designationfield 251 by using an R_(Y) designation screen 250 illustrated in FIG.37. In this example, an attitude of the end effector model EEM or theworkpiece model WM11 is expressed by a Z-Y-X system Euler's angle. Thus,a direction of a rotational axis of R_(Y) can be corrected by using thevalue of the R_(Z) designated in step S3402. The correction rotational Yaxis AXY which is a computed correction rotational axis is displayed inthe image display field 141 as described above. If the R_(Y) rotationangle is adjusted by displaying the rotational axis as mentioned above,a user can imagine how the end effector model EEM is changed, and caneasily perform setting.

R_(X) Designation Unit

Finally, in step S3404, a grip attitude R_(X) is designated. Herein, anR_(X) rotation angle is designated from an R_(X) rotation angledesignation field 261 by using an R_(X) designation screen 260illustrated in FIG. 38. The correction rotational X axis AXX indicatinga direction of the rotational axis of R_(X) is also displayed in theimage display field 141.

In the above-described way, a user can designate six position parameterssuch as X, Y, Z, R_(X), R_(Y), and R_(Z) regarding a grip position atwhich an end effector grips a workpiece. Particularly, since a positionparameter which can be adjusted is restricted for each screen, and arotational axis to be displayed is restricted to display of only arotational axis regarding a position parameter related to designation, auser performs setting according to such guidance, and can thussequentially define a necessary position parameter. As a result, it ispossible to smoothly perform designation work, difficult in the relatedart, for a grip position at which three-dimensional position andattitude are designated. When R_(X), R_(Y) and R_(Z) rotation angles ofEuler's angles are set, rotational axes thereof are computed, and aredisplayed as correction rotational axes in the image display field 141,and thus a user can easily imagine how rotational axes are moved bychanging position parameters which are being set.

If registration of a grip position is divided into a plurality of steps,and a position parameter which can be adjusted is restricted in eachstep, a user can easily imagine how a rotational axis is moved by movingthe position parameter. Particularly, a designation screen is providedin each step, and different position parameters are registered ondifferent designation screens. Thus, a position parameter to be set by auser can be presented without confusion, and it is possible to guidesetting work. In the example illustrated in FIGS. 36 to 38, adescription has been made of an example in which different designationscreens are provided, but the present invention is not limited to thisexample. For example, as illustrated in a position parameter designationscreen 270 in FIG. 39, a position parameter which can be designated maybe restricted in each step by using a common designation screen fordesignating a plurality of position parameters. In the exampleillustrated in FIG. 39, only X and Y coordinates can be designated froman X and Y coordinate designation field 271, and other positionparameters are grayed out so as not to be designated and selected. Ifdesignation of the X and Y coordinates is completed, and a “next” button272 is pressed, only separate position parameters, for example, a Zcoordinate and an R_(Z) rotation angle can be designated from a Zcoordinate and R_(Z) rotation angle designation field 273 in the nextstep, and other position parameters including the designated X and Ycoordinates are all grayed out or not displayed to be unselectable. Ifthe Z coordinate and the R_(Z) rotation angle are designated, and thenthe “next” button 272 is pressed, only further separate positionparameters, for example, R_(X) and R_(Y) rotation angles can bedesignated from an R_(X) and R_(Y) rotation angle designation field 274in the next step, and other position parameters cannot be selected. Withthis configuration, it is also possible to realize a guidance functionof presenting restricted position parameters which can be designated toa user, and sequentially setting the position parameters.

The number or division of steps for sequentially defining positionparameters is not limited to the above-described example. For example, aZ coordinate and an R_(Z) rotation angle may be set in separate steps,or an R_(X) rotation angle and an R_(Y) rotation angle may be set in thesame step. For example, as an example of dividing a step related to aposition into a plurality of steps, the Z-R_(Z) designation unit isformed of a Z designation unit for designating a Z coordinate and anR_(Z) designation unit for designating an R_(Z) rotation angle centeringon the Z axis. Alternatively, the R_(X)-R_(Y) designation unit may beformed of an R_(X) designation unit for designating an R_(X) rotationangle centering on the X axis and an R_(Y) designation unit fordesignating an R_(Y) rotation angle centering on the Y axis.Alternatively, the screens illustrated in FIGS. 37 and 38 may beintegrated into a single screen and used as an R_(X)-R_(Y) designationunit such that an R_(X) rotation angle and an R_(Y) rotation angle canbe designated.

In the above-described example, a description will be made of an examplein which a position and an attitude of an end effector model areadjusted in a state in which a workpiece model side is fixed, but thepresent invention is not limited to the example, and a position and anattitude of a workpiece model side may be adjusted in a state in whichan end effector model is fixed. Alternatively, positions and attitudesof both of an end effector model and a workpiece model may be adjusted.

In the above-described example, a description will be made of an examplein which, regarding the six position parameters such as X, Y, Z, R_(X),R_(Y), and R_(Z), X and Y coordinates are defined in the X-Y designationunit, a Z coordinate and an R_(Z) rotation angle are defined in theZ-R_(Z) designation unit, and R_(X) and R_(Y) rotation angles aredefined in the R_(X)-R_(Y) designation unit in this order, but an orderor a combination of defining the respective position parameters is notlimited to the above-described configuration. For example, an X-Y-Zdesignation unit, an R_(Z) designation unit, and an R_(X)-R_(Y)designation unit may be prepared, X, Y and Z coordinates of an endeffector model or a workpiece model may be designated in the X-Y-Zdesignation unit, then an R_(Z) rotation angle centering on the Z axismay be designated in the R_(Z) designation unit, and, finally, an R_(X)rotation angle centering on the X axis and an R_(Y) rotation anglecentering on the Y axis may be designated in the R_(X)-R_(Y) designationunit. In this case, for example, an X-Y-Z designation screen 280 asillustrated in FIG. 40 may be used as the X-Y-Z designation unit, and X,Y and Z coordinates may be designated from an X, Y and Z-coordinatedesignation field 281 with respect to the end effector model EEM or aworkpiece model displayed in a three-dimensional manner. As mentionedabove, an image for initially designating a grip position is not limitedto a plane image projected onto an XY plane as illustrated in FIG. 35B,and different projection directions may be used, and, for example,designation may be performed by using a perspective view WM11P obtainedthrough projection from an inclined direction as illustrated in FIG. 41.

The present invention is not limited to the method of designating a gripposition on an image projected onto a plane as in FIG. 35B, and a gripposition may be designated in a state without projection as illustratedin FIG. 35A. For example, in a state in which the workpiece model WM11is three-dimensionally displayed on a three-dimensional image viewer 290as illustrated in FIG. 42, a grip position is designated on the screenby using an operation unit formed of a pointing device such as a mouse.

Modification Examples

In the above-described example, a description will be made of proceduresof designating the six position parameters such as X, Y, Z, R_(X),R_(Y), and R_(Z) according to a Z-Y-X system Euler's angle, but thepresent invention is not limited to this aspect, and a position and anattitude of an end effector or the like may be defined as other aspectsof defining a position and an attitude of an end effector or aworkpiece, for example, an X-Y-Z system Euler's angle, a X-Z-Y systemEuler's angle, a Y-X-Z system Euler's angle, a Y-Z-X system Euler'sangle, a Z-X-Y system Euler's angle, an X-Y-X Euler's angle, an X-Z-XEuler's angle, a Y-X-Y system Euler's angle, an Y-Z-Y Euler's angle, aZ-X-Z system Euler's angle, an Z-Y-Z Euler's angle, or roll/pitch/yawangle expression, and a rotational axis/rotation angle expression.

In the above-described example, a description has been made of anexample in which only a correction rotational axis regarding a positionparameter to be designated is displayed as a correction rotational axis,but other correction rotational axes are not limited to a configurationin which the axes are completely not displayed. For example, the sameeffect can be achieved even if three orthogonal axes including othercorrection rotational axes are displayed, and a rotational axis relatedto a rotation angle to be designated is highlighted more than othercorrection rotational axes. As an example, a rotational axis related toa rotation angle which is being designated is displayed in boldface, andis highlighted through coloring or blinking, and, on the contrary, arotational axis which cannot be designated is grayed out or is displayedin a thin line, so that appearance thereof is distinct. Particularly, inthree-dimensional display of an image, if there are orthogonalcoordinate axes, a user can intuitively recognize three-dimensionaldisplay, and thus the user hardly confuses rotational axes bydifferentiating a target rotational axis from other rotational axeswhile display of three axes is maintained. The origin of when the threeaxes are displayed is preferably the center of rotation. For example,the origin is set to an intermediate position of a pair of claws whichis a grip position of the end effector model EEM as described above.

Grip Position Copying Function

A grip position copying function may be provided in which, when aplurality of grip positions are registered, a grip position which isalready registered is read, and a grip position is changed on the basisof the grip position so as to be registered as a new grip position.Generally, a plurality of grip positions are often registered for asingle workpiece. This is because, if a plurality of grip positions areregistered, an optimal solution can be selected from among a pluralityof grip solutions, and, thus, even in a case where an obtained gripsolution candidate interferes, if there are other grip solutioncandidates, there is a high probability that it is determined that gripis possible. In a case where a plurality of grip solutions areregistered, if grip registration is performed from the beginning everytime, much trouble is caused when the same grip position is registered,and thus the work is troublesome. Therefore, a grip position which isalready registered is copied, some position parameters set for the gripposition are changed, and thus a new grip position can be stored.Therefore, time and effort can be saved, and a plurality of grippositions can be easily registered. Similarly, an existing grip positionmay be read, and a position parameter may be corrected, overwritten, andstored.

Grip Position Copying Portion 8 d 8

The grip position copying function or a grip position editing functionis realized by using the grip position copying portion 8 d 8 illustratedin FIG. 26. The grip position copying portion 8 d 8 reads a gripposition of a workpiece model or an end effector model stored in thegrip position storage portion 9 b, and changes and registers positionparameters forming the grip position. If position parameters are alsosequentially registered according to guidance as described above when agrip position is changed by the grip position copying portion 8 d 8, itbecomes easier to understand which position parameter is preferablychanged in a corresponding step, and thus grip registration issimplified.

As described above, a fitting function may be added in which, when agrip position is designated, an end effector model is automaticallymoved to a grip position of a workpiece model. For example, a “fit”button 154 may be provided in the operation field 142 on the Z-R_(Z)designation screen 240 in FIG. 36, and the fitting function may beexecuted such that a Z coordinate can be automatically set.

Embodiment 5

An attitude or an angle which can be taken by a search model may berestricted in order to improve the accuracy of a three-dimensionalsearch. For example, it is considered that a position and an attitude ofa workpiece are detected through a three-dimensional search from aworkpiece group in which a plurality of plate-like workpieces WK9 asillustrated in FIG. 43 are loaded in bulk. In this case, respectivefaces of a workpiece model WM9 representing the workpieces WK9 in FIG.43 with three-dimensional CAD data are illustrated in FIGS. 44A to 44F.In other words, FIG. 44A is a plan view of the workpiece model WM9, FIG.44B is a bottom view thereof, FIG. 44C is a front view thereof, FIG. 44Dis a rear view thereof, FIG. 44E is a right side view, and FIG. 44F is aleft side view. Faces excluding repeated faces from the fundamentaldirection images are registered as search models. Herein, it is assumedthat four faces in FIGS. 44A, 44B, 44C and 44E are registered. In thisstate, point group data having three-dimensional information is acquiredby imaging the bulk workpiece group, and a three-dimensional search isperformed. As a result, wrong detection may occur as if the workpiece ispresent at a vertical attitude as indicated by X in FIG. 45. Actually, athin plate-like workpiece is scarcely upright, and thus a plate-likeworkpiece is substantially linear or rectangular in a front view asillustrated in FIG. 44C. Therefore, there is a case where a shape of apart of the workpiece is in common with a shape of another part forminga surface of the workpiece, for example, the shape illustrated in FIG.44A or 44B, and thus wrong detection occurs.

In contrast, an attitude which can be taken by a workpiece may berestricted, and a three-dimensional search may be performed, but it isdifficult to set a condition for performing attitude restriction. Forexample, there is a method in which all attitudes of a workpiece can beexpressed by defining rotation angle ranges of the Z axis, the Y axis,and the Z axis by using a Z-Y-Z system Euler's angle as illustrated inFIG. 46, but, in this method, it is not easy to define a rotation angleabout each axis such that an attitude range desired by a user isobtained.

Therefore, in Embodiment 5, a user can easily set an attitude which canbe taken by a workpiece without using such a troublesome concept.Specifically, when a search model is registered, an attitude which isscarcely taken in bulk is excluded, and thus a face related to theattitude is not detected. Such setting is performed by using the searchmodel registration portion 8 g illustrated in FIG. 6. The search modelregistration portion 8 g selects a fundamental direction imageregistered as a search model for performing a three-dimensional searchfor specifying a position and an attitude of each workpiece from amongsome fundamental direction images with respect to a plurality ofworkpiece groups loaded in bulk. Regarding a difference between thesearch model registration portion 8 g and the fundamental directionimage selection portion 8 e, the fundamental direction image selectionportion 8 e selects one of fundamental direction images having the sameviewing way. In contrast, the search model registration portion 8 gexcludes a fundamental direction image which is not desired to be asearch target during a three-dimensional search from a search modelregistration target. Preferably, in a state in which the fundamentaldirection image selection portion 8 e excludes a fundamental directionimage having the same viewing way, the search model registration portion8 g further excludes an unnecessary fundamental direction image from asearch model registration target. Alternatively, a fundamental directionimage desired to be searched is selected. Consequently, a user selectsfundamental direction images which are narrowed to search modelregistration candidates in advance, and can thus easily perform searchmodel registration work by reducing the number of candidates.

Search Model Registration Screen 130B

Here, FIG. 47 illustrates an example of a search model registrationscreen 130B on which a search model is registered for the workpiece inFIG. 43 described above. On the search model registration screen 130Billustrated in FIG. 47, the fundamental direction image selectionportion 8 e excludes two fundamental direction images in FIGS. 44D and44F having the same viewing way, in the examples illustrated in FIGS.44A to 44F, from six fundamental direction images corresponding to sixdrawings generated by the fundamental direction image generation portion8 e′ with respect to the workpiece model WM9 corresponding to theworkpieces WK9 in FIG. 43, and displays the remaining four fundamentaldirection images in the six-drawing display region 3 a of the displayunit in this state. In this state, as an aspect of the search modelregistration portion 8 g setting whether or not a search model is to beregistered, a selection checkbox 131B such as “register” displayed ineach fundamental direction image on the search model registration screen130B is provided. If a user checks the selection checkbox 131B, acorresponding fundamental direction image is registered as the searchmodel, but, conversely, if the user does not check the selectioncheckbox 131B, a corresponding fundamental direction image is excludedfrom a search model registration target. A user can exclude an attitudewhich is scarcely taken in an actual bulk state while viewing eachfundamental direction image displayed in the six-drawing display region3 a. In the example illustrated in FIG. 47, the selection checkboxes131B of the model C and the model D are not checked, and attitudes atwhich such side faces are viewed, that is, attitudes at which thetabular workpiece is upright can be excluded from three-dimensionalsearch targets. Consequently, a user can exclude an image on which asearch is not required to be performed or select only a necessary imagewhile viewing a fundamental direction image indicating an attitude of aworkpiece without performing work such as troublesome angle computationor range designation for the workpiece, and can easily put a restrictionon an attitude of a workpiece which is a three-dimensional searchtarget.

The search model registration portion 8 g is not limited to an aspect inwhich a user manually selects a search model registration target or asearch model exclusion target, and may automatically extract and excludea fundamental direction image of an attitude which is scarcely taken bycalculating a shape or the centroid of a workpiece model.

Alternatively, a combination of automatic calculation and manualselection may be used, and, for example, as illustrated in FIG. 47,there may be a configuration in which the selection checkbox 131B isdisplayed not to be checked as an initial state with respect to afundamental direction image of an attitude which is determined as beingscarcely taken through calculation while displaying all fundamentaldirection images selected by the fundamental direction image selectionportion 8 e on the search model registration screen 130B. A user canmanually correct selection as necessary while referring to an automaticdetermination result in the search model registration portion 8 g, andcan more reliably register a search model by pressing an OK button.

In FIG. 47, a description has been made of the example in whichselection in the fundamental direction image selection portion 8 e isperformed through automatic calculation, but automatic determination andmanual selection in the fundamental direction image selection portionmay be combined with each other in the same manner. For example, as aresult of determination in the fundamental direction image selectionportion, one of fundamental direction images determined as having thesame viewing way is grayed out and is displayed in the six-drawingdisplay region, and thus a user can check a result of automaticdetermination in the fundamental direction image selection portion. In acase where a determination result is wrong, the user may manually selectthe fundamental direction image so as to remain the fundamentaldirection image as a fundamental direction image or conversely excludethe fundamental direction image. As mentioned above, a result ofautomatic calculation is displayed and is checked by a user, and thus itis possible to further increase the accuracy of image selection. A statein which selection/non-selection is set in advance on the basis of anautomatic determination result is used as an initial state. Thus, if aresult of selection based on automatic determination is correct, a userpresses an OK button so as to approve the selection, so that time andeffort on the user side can be minimized.

The purpose of excluding an unnecessary face from a three-dimensionalsearch target is also applicable to states other than a workpieceupright state. For example, in addition to an aspect in which workpiecesare stacked completely randomly, in a state in which an input image of aworkpiece group is given in an aspect in which a specific face isexposed, it is possible to prevent wrong detection by excluding facesnot exposed. Particularly, this is effective to a workpiece of whichshapes of front and rear faces are similar to each other, and thus therear face is easily wrongly detected. As mentioned above, if a searchmodel of the rear face which cannot be actually viewed has only to beexcluded from a three-dimensional search target, it is possible toachieve the same effect as restricting an attitude of a workpiece suchthat a rear face side of the workpiece is not detected.

Procedures of Registering Search Model

A description will be made of procedures of registering a search modelin which an attitude restriction is provided with reference to aflowchart of FIG. 48. Herein, a description will be made of proceduresof registering three-dimensional CAD data of a workpiece as a searchmodel.

First, in step S4801, three-dimensional CAD data of a workpiece is read.Next, in step S4802, the center of a circumscribing cuboid of thethree-dimensional CAD data model is corrected to the origin of thethree-dimensional CAD data. In step S4803, height images viewed fromrespective directions of “top”, “bottom”, “left”, “right”, “front”, and“rear” are generated. Here, in a case where a height image is generatedon the basis of the three-dimensional CAD data, the height image isgenerated such that the origin of CAD is the center of the height image.In step S4804, a height image having the same viewing way is deletedfrom the generated height images.

In step S4805, a search model used for a three-dimensional search isselected from among the remaining height images. Herein, the searchmodel selection portion 8 i excludes an unnecessary fundamentaldirection image, and selects a necessary fundamental direction image.

Finally, in step S4806, the selected height image is registered as asearch model. Herein, the search model registration portion 8 gregisters the search model. In the above-described way, a fundamentaldirection image of an unnecessary attitude can be excluded, and thus asearch model can be registered in a state in which an attituderestriction is substantially provided.

Embodiment 6

A description has been made of an example of restricting athree-dimensional search target by using an image instead of a numericalvalue. However, the present invention is not limited to this aspect, andan attitude may be restricted by using an inclined angle or a rotationangle of a search model instead thereof or in addition thereto.Consequently, a three-dimensional search condition can be appropriatelyset according to a state in which workpieces are actually stacked, andthus it is possible to reduce wrong detection in a search. An example ofan inclined angle/rotation angle setting screen 160 on which an attituderestriction using an inclined angle and a rotation angle is performed isillustrated in FIG. 49 as Embodiment 6. The inclined angle/rotationangle setting screen 160 illustrated in FIG. 49 includes an attituderestriction search model selection field 161 for selecting a searchmodel on which an attitude restriction is performed, an inclined angleupper limit setting field 162 for designating an upper limit of anallowable inclined angle, and a rotation angle range setting field 163for defining an allowable rotation angle range.

Inclined Angle/Rotation Angle Setting Screen 160

On the inclined angle/rotation angle setting screen 160, designation isperformed by using an “inclined angle” and a “rotation angle” from anattitude during registration of a search model as easily understandableparameters without having professional knowledge instead of a difficultmethod of designating a three-dimensional attitude by using, forexample, R_(X), R_(Y), and R_(Z) indicating rotation angles aboutcoordinate axes for a workpiece. Herein, an inclined angle and arotation angle are designated from a state in which an attitude in afundamental direction image registered as a search model is used as aregistration attitude (details thereof will be described later).

Specifically, on the inclined angle/rotation angle setting screen 160 inFIG. 49, a search model on which an attitude restriction is imposed isselected in the attitude restriction search model selection field 161.In the example illustrated in FIG. 49, the search model of the model Aregistered on the search model registration screen 130B in FIG. 47 isselected.

An inclined angle within a predetermined range which is output as aresult of a three-dimensional search with respect to the attitude duringregistration is set in the inclined angle upper limit setting field 162.

A reference angle and an angle range are set in the rotation angle rangesetting field 163. First, a rotation angle for the attitude duringregistration is input to a reference angle setting field. The rotationangle set here is a reference angle when a rotation angle range isdesignated. A rotation angle range which is output as a result of athree-dimensional search with respect to the reference angle set in thereference angle setting field is set in a range setting field.

Attitude Restriction Method Using Inclined Angle and Rotation Angle

Here, a description will be made of a method of obtaining an inclinedangle and a rotation angle on the basis of a three-dimensional attitudeof a search model. In a method of restricting a face to be registered asa search model in the three-dimensional search, a form of automaticallyimposing an attitude restriction is obtained. On the other hand, in acase where an attitude restriction is imposed on the face registered asa search model, an attitude registered for the face is used.

Herein, an attitude restriction is performed with two angles such as an“inclined angle” for a registered attitude and a “rotation angle”directly viewed from the top for the registered attitude. If only thetwo angles are used, the concept thereof is easily understandable, andan angle restriction can be performed relatively simply.

Inclined Angle

First, an inclined angle is defined as an angle for a Z verticaldirection of an attitude of a workpiece model when registered as asearch model as illustrated in FIGS. 50A to 50C. FIG. 50A illustrates anattitude of a workpiece model WMR when a search model is registered anda Z axis defined as a vertical direction of the workpiece model. Incontrast, an attitude of a workpiece WKI and a Z′ axis defined as avertical direction of the workpiece WKI included in an input image arein a state as illustrated in FIG. 50B. In this state, an “inclinedangle” is defined as an inclined angle between the Z axis and the Z′axis, that is, an inclination of the workpiece in the verticaldirection, as illustrated in FIG. 50C. Consequently, the inclined anglecan be represented by a single angle such as an inclination relative toa registered state of the search model regardless of an inclineddirection. It is possible to obtain an advantage that a user canconceptually easily understand an inclined angle.

Rotation Angle

On the other hand, a rotation angle is a defined as a rotation anglewhen viewed from a vertical direction during registration, that is, theZ axis. Here, the rotation angle will be described with reference toFIGS. 51A to 51C. Among these figures, FIG. 51A illustrates a workpiecemodel WMR when a search model is registered, and a Y axis defining an XYplane. In contrast, an attitude of a workpiece WKI and a Y′ axisdefining an XY plane included in an input image are in a state asillustrated in FIG. 51B. In this state, a “rotation angle” is defined asa rotation angle when viewed from the vertical direction, that is, the Zaxis, as illustrated in FIG. 51C. In this example, the rotation angle isdefined as an angle of the Y axis viewed from the Z axis direction witha coordinate axis in the Y axis direction defining a workpiece model ora workpiece as a reference. With this definition, it is possible toobtain an advantage that a user can easily understand a rotation anglein the same concept as an angle in a two-dimensional pattern search ofthe related art.

As a method of defining the origin and XYZ axes for a workpiece, a knownalgorithm may be used. For example, as illustrated in FIG. 8, a figurecircumscribing a workpiece, for example, a cuboid or a sphere iscalculated on the basis of shape information of a workpiece model or theworkpiece included in an input image, the centroid thereof is calculatedto be used as the origin of the workpiece, and XYZ axes are defined.Regarding a method of defining XYZ axes, in a case where CAD data isused, directions of coordinate axes of original CAD data may be used asdirections of the respective XYZ axes. In a case of using actuallymeasured data obtained by actually three-dimensionally imaging aworkpiece, a vertical direction of the actually measured data may beused as a Z axis, an upper direction of the actually measured data in aplan view may be used as a Y axis, and a rightward direction of theactually measured data in a plan view may be used as an X axis.Regarding the origin, the centroid or a workpiece model or a centercoordinate of CAD data may be used as the origin.

Procedures of Obtaining Inclined Angle and Rotation Angle on the Basisof Three-Dimensional Attitude

Here, a description will be made of procedures obtaining an inclinedangle and a rotation angle on the basis of a three-dimensional attitudewith reference to a flowchart of FIG. 52. First, in step S5201, aninclined angle is obtained. For example, in a case where an attitude ofthe workpiece WKI included in an input image is as in FIG. 53B withrespect to the workpiece model WMR when registered in a state asillustrated in FIG. 53A, the inclined angle is obtained as aninclination of the Z axis in the vertical direction as illustrated inFIG. 53C.

Next, in step S5202, the input image is three-dimensionally rotated suchthat the Z′ axis matches the Z axis. In other words, the inclination isremoved. For example, as illustrated in FIG. 54A, the Z′ axis is rotatedto overlap the Z axis with an outer product vector VP between a vectorof the Z′ axis and a vector of the Z axis as a rotational axis withrespect to the input image WKI in FIG. 53C as illustrated in FIG. 54A.As a result, a rotated input image WKI′ is obtained as in FIG. 54B, andthe inclination is removed.

Finally, in step S5203, a rotation angle is obtained. For example, a Yaxis of the workpiece model WMR when registered, illustrated in FIG. 55Aand a Y′ axis of the input image WKI′ in which the inclination isremoved as illustrated in FIG. 55B, are obtained. In this state, asillustrated in FIG. 55C, an angle between the Y axis and the Y′ axiswhen viewed from the vertical direction, that is, the Z axis is obtainedas a rotation angle. In the above-described way, an inclined angle and arotation angle may be obtained on the basis of a three-dimensionalattitude in an input image.

Setting of Upper Limit of Inclined Angle

As an example of setting an inclined angle upper limit on the inclinedangle/rotation angle setting screen 160 in FIG. 49, for example, in acase of a metal workpiece with high gloss, a range in whichthree-dimensional measurement cannot be performed is widened as a faceis inclined. If a three-dimensional search is performed in this state, arisk that wrong detection occurs in the three-dimensional search isincreased. Therefore, too inclined attitude in which the risk of wrongdetection is equal to or more than a predetermined level is excludedfrom a three-dimensional search target, and thus it is possible to leaveonly a search result with high reliability. For example, if aninclination is equal to or more than 40 degrees, an inclined angle upperlimit is set to 40 degrees for a workpiece in which wrong detectioneasily occurs, and thus a search result in which an inclination is 40degrees or higher can be excluded.

Setting of Upper Limit of Rotation Angle Range

On the other hand, as an example of setting a rotation angle range onthe inclined angle/rotation angle setting screen 160 in FIG. 49, since,for example, the model A or the model B in FIG. 47 described above hasmany similar portions in the whole shape characteristics even in a stateof being rotated by 180 degrees, if the number of locations wherethree-dimensional measurement is not performed due to the influence ofmultiple reflection or the like is increased, a rotation attitude may bewrongly detected as an attitude rotated by 180 degrees. In a case wherereal workpieces are stacked only in a predetermined direction, a resultin an opposite direction can be excluded by using such an attituderestriction. For example, in a case where a rotation angle range is setto ±30 degrees with respect to a reference angle, only a search resultwithin the angle range is detected.

End Effector Attachment Position Setting Screen 170

The above method of defining a position and an attitude using an Euler'sangle may be used not only for defining a position and an attitude atwhich a workpiece model is gripped by an end effector model but also fordefining a position at which an end effector is attached to the tip ofthe arm portion of the robot. Here, a description will be made ofprocedures of setting a position at which an end effector is attached tothe tip of the arm portion of the robot. Setting of an attachmentposition of the end effector is performed on an end effector attachmentposition setting screen 170 as illustrated in FIG. 56. The end effectorattachment position setting screen 170 illustrated in FIG. 56 includesan image display field 141 and an operation field 142. An end effectormodel is displayed in the image display field 141. Three-dimensional CADdata which is created in advance is read as the end effector model.Alternatively, the end effector model may be formed of a basic figuresuch as a cuboid or a cylinder simulating an end effector so as to bedisplayed.

An end effector attachment position setting field 171 for defining anattachment position of an end effector attached to the tip of the armportion is provided in the operation field 142. End effector attachmentposition parameters set in the end effector attachment position settingfield 171 include attachment position parameters (X, Y, and Z) andattachment attitude parameters (R_(X), R_(Y), and R_(Z)) of the endeffector model. For example, the parameters are defined for the centerof the flange surface FLS at the tip of the arm portion by using anEuler's angle.

A positional relationship between a part (for example, a pair of clawsprovided at the tip) of an end effector model gripping a workpiece modeland a gripped workpiece is registered on the grip position registrationscreen of the workpiece model illustrated in FIG. 14 or the likedescribed above. In contrast, an attachment position at which the endeffector model is attached to the tip of the arm portion is registeredon the end effector attachment position setting screen 170. Therefore, aworkpiece model or the like is not required to be displayed, and the endeffector model and the flange surface FLS are displayed in the imagedisplay field 141.

Grip Position Designation Screen 180

In contrast, as illustrated in FIG. 57, an end effector model and aworkpiece model are displayed, and a location gripping the workpiecemodel is designated, on a grip position designation screen 180 fordesignating a grip position of the workpiece model. The grip positiondesignation screen 180 illustrated in FIG. 57 includes an image displayfield 141 and an operation field 142, and a grip target workpiece isselected, and the workpiece is drawn in a three-dimensional image and isdisplayed at an attitude at which a gripped face is directed toward asurface, in the image display field 141. An end effector model isfurther displayed, and is adjusted to a position and an attitude atwhich the workpiece model is gripped. A grip position designation field181 for designating a grip position is set in the operation field 142. Auser adjusts position parameters in the grip position designation field181 such that a relative relationship between the workpiece model andthe end effector model is accurate. Alternatively, a position or anattitude is adjusted on the screen by dragging the end effector model orthe workpiece model displayed in the image display field 141. A positionparameter adjusted on the screen is reflected in the grip positiondesignation field 181. A position parameter in the grip positiondesignation field 181 may be expressed by using the above-describedEuler's angle.

Plural-Grip Position Selection Screen 190

Regarding registration of a grip position, a single location is notdesignated for a single face (for example, a fundamental directionimage) of a certain workpiece, and a plurality of locations may bedesignated. Grip positions registered at a plurality of locations may bechecked with an image. For example, FIG. 58 illustrates an example of aplural-grip position selection screen 190. The plural-grip positionselection screen 190 also includes the image display field 141 and theoperation field 142. The operation field 142 is provided with a faceselection field 191 for selecting a face of a workpiece model for whicha grip position is defined and a grip position list display 192 in whichgrip positions set for a face selected in the face selection field 191are displayed in a list form. In this example, a face C is selected inthe face selection field 191, and grips 1 to 3 as grip positions set forthe face C are displayed in a list form in the grip position listdisplay 192. If, among the grips, the grip 2 is selected, a scene inwhich the end effector model grips the workpiece model at a gripposition and a grip attitude registered as the grip 2 is displayed inthe image display field 141. In the above-described way, a user canselect and check a plurality of registered grip positions. A registeredgrip position may be corrected and updated as necessary. Also herein, agrip attitude may be displayed by using the above-described Euler'sangle. In the above-described way, the concept of the Euler's anglewhich is difficult for a user to understand is displayed to be visuallyunderstandable without deep understanding thereof while registration isperformed by using the Euler's angle, a rotational axis is corrected anddisplayed, and a parameter which can be adjusted is restricted due tothe guidance function. Therefore, it is possible to perform setting workby reducing confusion.

Function of Correcting End Effector Attachment Position

As described above, an end effector is attached to the tip of the armportion of the robot. On the other hand, registration (teaching) of thegrip position or grip determination (simulation) of whether or not anend effector model can grip a workpiece model at registered position andattitude is performed on a virtual three-dimensional space of the robotsetting apparatus side (robot vision side) without using a real endeffector. Conversion from a coordinate position of a vision space whichis a virtual three-dimensional space into a coordinate position of arobot space which is a real space is performed by the conversion portion8 x on the basis of calibration information obtained by the calibrationportion 8 w in FIG. 17.

However, in a case where an attachment state of an end effector modelvirtually set on the robot setting apparatus side is different from anattachment state of the end effector EET of the real robot RBT, acorrespondence relationship between the vision space and the robot spacedefined in the calibration information is not maintained, and thusdeviation occurs when a workpiece is to be gripped by using the realrobot RBT.

Here, FIG. 59 illustrates a positional relationship between an endeffector model and the flange surface FLS. FIG. 59 illustrates theorigin O_(F) of the flange surface FLS at the tip of the arm portion ofthe robot, and the center O_(E) of the end effector model EEM as aposition of the end effector model EEM. A position of the end effectormodel EEM is set as a position (X, Y, and Z) and an attitude (R_(X),R_(Y), and R_(Z)) for the flange surface FLS at the tip of the armportion of the robot on the robot setting apparatus side. In otherwords, in the example illustrated in FIG. 59, the position (X, Y, and Z)and the attitude (R_(X), R_(Y), and R_(Z)) of the center O_(E) of theend effector model EEM viewed from the origin O_(F) of the flangesurface FLS are set. Here, if there is an error in a real attachmentstate of an end effector, deviation may occur in the position of the endeffector model EEM for the origin of the flange surface FLS. As aresult, gripping is performed at a deviated position on the robotsetting apparatus side and the robot side due to the attachment state ofthe end effector.

Therefore, in the present embodiment, a function of correcting an endeffector attachment position is provided to reflect an attachment stateof a real end effector on the robot setting apparatus side, and thussuch deviation or an error does not occur. The function of correcting anend effector attachment position is realized by the end effectorattachment position correction portion 8 y illustrated in FIG. 17.Specifically, a three-dimensional search is performed on actuallymeasured data obtained by imaging a real end effector and performingthree-dimensional measurement by using three-dimensional CAD data whichis an end effector model used for registration of a grip position of aworkpiece or for interference determination when the end effector isdisposed and which virtually expresses a three-dimensional shape of theend effector, so as to acquire a position and an attitude of a realattachment position. Therefore, such an error is detected, and thussetting of an end effector attachment position on the robot settingapparatus side is corrected. As mentioned above, a user can performmanual adjustment while checking deviation between a point groupobtained by three-dimensionally measuring a real end effector and anattachment state of an end effector model defined on a virtualthree-dimensional space on the robot setting apparatus side in the imagedisplay region of the display unit.

Procedures of Automatically Correcting Deviation

Here, a description will be made of a correction function ofautomatically correcting an error between a real end effector and an endeffector model formed of three-dimensional CAD data with reference to aflowchart of FIG. 60. It is assumed that calibration of converting acoordinate position of a vision space which is a virtual space into acoordinate position of a robot space which is a real space is performedby the calibration portion 8 w illustrated in FIG. 17 in advance, andcalibration information is held in the storage unit 9. It is alsoassumed that an end effector model is prepared by usingthree-dimensional CAD data.

End Effector Imaging Screen 330

First, in step S6001, preparation for imaging a real end effector in thesensor unit is made. Specifically, the robot is operated to move the endeffector such that the end effector can be imaged by a three-dimensionalcamera which is an aspect of the sensor unit. Next, in step S6002,three-dimensional measurement is performed on the end effector.

For example, an end effector imaging screen 330 illustrated in FIG. 61is used as an aspect of an end effector imaging unit for imaging thereal end effector in the sensor unit. The end effector imaging screen330 includes an image display field 141 and an operation field 142. Theend effector EET attached to the flange portion at the tip of the armportion ARM imaged by the sensor unit 2 is displayed in the imagedisplay field 141 in real time. The operation field 142 is provided withan end effector position designation field 331 showing a position and anattitude of the end effector EET. A user manually operates the robotsuch that the camera forming the sensor unit can image the end effectorEET of the robot on the end effector imaging screen 330. In this case,an imaging target is not the claw portion of the end effector EETgripping a workpiece but an attachment state with the arm portion.Therefore, preferably, imaging is performed such that an area of the endeffector EET is increased. Preferably, the entire image of the endeffector EET is captured by the camera at a downward attitude at whichthe end effector EET easily grips a normal workpiece and also at ahorizontal attitude. In the above-described way, the user positions theattitude of the end effector EET and then presses an “imaging” button332. Consequently, a three-dimensional captured image of the endeffector is acquired. In other words, three-dimensional measurement isperformed on the end effector.

Next, in step S6003, position-attitude A of the flange portion in arobot coordinate system is acquired. Here, the position-attitude A inthe robot coordinate system is assumed to be, for example, a positionand an attitude in the robot coordinate system of the flange portion FLSto which the imaged end effector is attached. Orders of the processes instep S6003 and step S6002 may be replaced with each other.

In step S6004, the position-attitude A of the flange portion in therobot coordinate system is converted into position-attitude B on thevision space. Herein, the conversion portion 8 x in FIG. 17 performsconversion between the real space and the virtual space on the basis ofthe calibration information for coordinate conversion between the robotspace and the vision space, obtained by performing calibration inadvance.

In step S6005, a three-dimensional search is performed such thatposition-attitude C of the end effector on the vision space is detected.Herein, a three-dimensional search is performed on the three-dimensionalmeasured data obtained in step S6002 by using three-dimensional CAD dataused for the end effector model as a search model. A three-dimensionalsearch method may employ a known algorithm as appropriate. Consequently,a position and an attitude of the end effector on the vision space aredetected.

In step S6006, relative position-attitude of the position-attitude C ofthe end effector for the position-attitude B of the flange portion FLSon the vision space is calculated. A coordinate position obtained heregives a position and an attitude of the end effector model which areaccurate for the flange portion FLS, in other words, in which positiondeviation or the like is taken into consideration.

Finally, in step S6007, the obtained position and the attitude arereflected in end effector setting on the vision side. In other words,the obtained position and the attitude are reflected in setting of aposition and an attitude of the end effector model on the vision side.In the above-described way, an attachment state of a real end effectorcan be automatically reflected on the vision side.

Procedures of Manually Correcting Deviation

The above description relates to the procedures of automaticallycorrecting deviation between the robot space and the vision space.However, the present invention is not limited to a configuration ofautomatically correcting deviation, and deviation may be manuallycorrected. Next, a description will be made of procedures of manuallycorrecting deviation with reference to a flowchart of FIG. 150. Here,processes in steps S15001 to S15004 are the same as the processes insteps S6001 to S6004 in FIG. 60 related to the above-described automaticcorrection, and detailed description thereof will be omitted. That is,in step S15001, the robot is operated to move the end effector such thatthe end effector of the robot can be imaged by a three-dimensionalcamera, in step S15002, three-dimensional measurement is performed onthe end effector, in step S15003, position-attitude A of the flangeportion in a robot coordinate system is acquired, and, in step S15004,the position-attitude A of the flange portion in the robot coordinatesystem is converted into position-attitude B on the vision space.

In step S15005, position-attitude D of the end effector is calculated onthe basis of the end effector setting from the position-attitude B ofthe flange portion on the vision space. Herein, the position-attitude Dof the end effector is obtained on the basis of information regardingthe position-attitude B of the flange portion at the tip of the robot onthe vision space, and the position-attitude of the end effector for theflange portion set in the end effector setting.

Next, in step S15006, a point group obtained through three-dimensionalmeasurement and CAD display of the position-attitude D of the endeffector are displayed in a superimposed manner. Herein, the endeffector model EEM which is three-dimensional CAD data of the endeffector is displayed at a position of the position-attitude D of theend effector obtained in step S15005, and a point group PC obtainedthrough three-dimensional measurement is displayed to be superimposed onthe real end effector. FIGS. 151 and 152 illustrate examples of adeviation correction screen on which such superimposed display isperformed.

In step S15007, it is determined whether or not there is deviation whichcannot be ignored between the point group and the CAD display. In a casewhere there is deviation which cannot be ignored, the flow proceeds tostep S15008, and the end effector setting is changed such that thedeviation is corrected, and then the flow returns to step S15005 suchthat the above-described processes are repeatedly performed. The endeffector setting includes a position and an attitude of the end effectorfor the flange portion. On the other hand, in a case where there is nodeviation, the process is finished.

Deviation Correction Unit

FIG. 151 illustrates an example of a deviation correction screen as anaspect of a deviation correction unit performing deviation correction.FIG. 151 illustrates a case where there is deviation which cannot beignored. The point group PC is displayed to be superimposed on the endeffector model EEM in the image display field 141 on a deviationcorrection screen 840. As illustrated in FIG. 151, it can be seen thatthe point group PC obtained by actually three-dimensionally measuringthe end effector and indicated by white points is deviated in the leftobliquely downward direction relative to the end effector model EEM.

The operation field 142 is provided with an end effector setting field841, and a position (X, Y, and Z) and an attitude (R_(X), R_(Y), andR_(Z)) of the end effector for the flange portion are defined. Herein,long coordinate axes of XYZ illustrated on the right part in FIG. 151indicate the origin O_(F) of the flange portion FLS which is the robottip. Short coordinate axes of XYZ illustrated on the left part indicatethe origin O_(E) of three-dimensional CAD data of the end effector modelEEM for the robot tip (flange portion). The user adjusts a position andan attitude of the end effector model EEM while checking the endeffector model EEM and the point group PC displayed in the image displayfield 141. For example, a position and an attitude of the end effectormodel EEM are designated as a numerical value in the end effectorsetting field 841 of the operation field 142, or the end effector modelEEM is dragged and moved on the image display field 141, so that endeffector model EEM is adjusted to be superimposed on the point group PC.

FIG. 152 illustrates an example in which the end effector model EEMmatches the point group PC in the above-described way. In this example,in the end effector setting field 841, a position in the Z direction isset to 90 mm, and thus matching occurs by offsetting the end effectormodel EEM in the Z direction. After the position and the attitude of theend effector model are adjusted, if an “OK” button 842 is pressed, theend effector setting is updated. Consequently, a position and anattitude on the vision side can be adjusted according to an attachmentstate of a real end effector.

The above-described manual deviation correction may be performed incombination with automatic correction. Such an example will be describedwith reference to a deviation correction screen 850 in FIG. 153. In thedeviation correction screen 850 illustrated in FIG. 153, the endeffector setting field 841 for manually adjusting a position and anattitude of the end effector model EEM is provided in the operationfield 142, and an “automatic correction” button 843 is provided on alower part. If the “automatic correction” button 843 is pressed, athree-dimensional search is performed internally, and a position and anattitude of the end effector model EEM are automatically corrected tomatch the point group PC. For example, automatic deviation correctionmay be performed or an automatic correction result may be manuallyfinely adjusted while checking states of the end effector model EEM andthe point group PC displayed to be superimposed on each other in theimage display field 141. Particularly, in a case where the automaticcorrection function cannot be executed well since metal with high glossis used for an end effector, and thus the point group PC is notaccurately detected due to reflected light, for example, in a case wherea three-dimensional search fails, manual adjustment can cope with thiscase. In the above-described way, a user can manually correct deviationbetween the robot space and the vision space.

In the related art, in a case where an attachment member such as aconnector can be assembled with an attachment portion between the flangeportion and the end effector, but an offset amount thereof is forgottento be set on the vision side, there is a problem in that a position oran attitude is deviated. In this case, if there is no function ofchecking deviation or function of correcting deviation, there is aproblem in that gripping cannot be accurately performed when a workpieceis gripped by the robot during an actual operation, or collision occurs.Causes of such errors are various, and, in the related art, it takeslots of time and effort to investigate cause and debug. In contrast,according to the present embodiment, deviation can be manually orautomatically corrected, and thus it is possible to easily performflexible adjustment according to an attachment state of a real endeffector.

In the example illustrated in FIG. 59, a position of the end effectormodel EEM is set to a central position of three-dimensional CAD data,but this is only an example, and other positions may be defined. Forexample, in FIG. 59, the center of the cuboid circumscribingthree-dimensional CAD data is used, but an end portion of thecircumscribing cuboid or the origin of three-dimensional CAD data as aninput source may be used as a reference.

Procedures of Registering Actually Measured Data in Search Model

The above description relates to procedures in a case wherethree-dimensional CAD data is registered as a search model. However, thepresent invention is not limited to three-dimensional CAD data as asearch model as described above, and, for example, actually measureddata obtained by actually imaging a workpiece in the sensor unit may beregistered as a search model. Here, in step S2701 in FIG. 27, adescription will be made of procedures of registering actually measureddata as a search model with reference to a flowchart of FIG. 62.

First, in step S6201, a face of a workpiece desired to be registered isdirected upward, the workpiece is placed on a flat surface, andthree-dimensional measurement is performed by the sensor unit.

Next, in step S6202, obtained actually measured data is registered as asearch model.

Finally, in step S6203, it is determined whether or not the number ofsearch models required for a three-dimensional search is registered,and, in a case where the number of search models required for athree-dimensional search is not registered, the flow returns to stepS6201, the processes are repeatedly performed, and, in a case where thenumber of search models required for a three-dimensional search isregistered, the process is finished. Details of the procedures will bedescribed later with reference to FIGS. 102 to 106.

First Procedures During Actual Operation

In a state in which necessary setting work is finished in theabove-described way, a picking operation is performed on a workpiecegroup actually loaded in bulk. Here, a description will be made ofprocedures of determining whether or not a workpiece can be grippedduring an actual operation, that is, whether or not there is a gripsolution for each detected workpiece in a state in which the searchmodel is registered according to the procedures illustrated in FIG. 28,with reference to a flowchart of FIG. 63. Herein, the calculation unit10 in FIG. 6 determines the presence or absence of a grip solution.

First, in step S6301, three-dimensional measurement starts to beperformed on bulk workpieces. Herein, the three-dimensional measurementis performed by imaging a bulk workpiece group in the sensor unit, andthus a three-dimensional shape having height information is acquired.

Next, in step S6302, a three-dimensional search is performed on theobtained three-dimensional shape of the workpiece group by using aworkpiece model, and a position and an attitude of each workpiece aredetected.

Next, in step S6303, with respect to a single detected workpiece, aposition and an attitude at which an end effector is to be disposed arecomputed on the basis of a position of the workpiece and a grip attitudeof the workpiece registered during setting.

Next, in step S6304, interference determination of whether or not theend effector interferes with a peripheral object at the computedposition is performed by using an end effector model.

In step S6305, it is determined whether or not the end effectorinterferes, and, in a case where the end effector does not interfere, itis determined that there is a grip solution for this workpiece, and theprocess is finished.

On the other hand, in a case where it is determined that the endeffector interferes, the flow proceeds to step S6306, and it isdetermined whether or not there are other grip positions registered forthis workpiece. In a case where other grip positions are registered, theflow returns to step S6303, and the processes are repeatedly performedon the grip positions.

On the other hand, in a case where other grip positions are notregistered, the flow proceeds to step S6307, and it is determinedwhether or not there are other detected workpieces. In a case wherethere are other workpieces, the flow returns to step S6303, and theprocesses are repeatedly performed on the workpieces instead of theworkpiece. In a case where there are no other workpieces, it isdetermined that there is no grip solution, and the process is finished.

In the above-described way, the calculation unit 10 in FIG. 6 determinesthe presence or absence of a grip solution in which a workpiece can begripped. In a case where a grip solution is obtained, an instruction isgiven to the robot controller 6 such that the workpiece is gripped by anend effector at a determined grip position. Thus, the robot controller 6controls the end effector to pick the workpiece as instructed.

In the above-described procedures, if a grip solution is obtained withany workpiece, a process of examining a grip position is finished atthis time, and control is performed such that the workpiece is grippedat a grip position corresponding to the obtained grip solution. However,the present invention is not limited to this method, and, for example,there may be a configuration in which all grip positions at which gripis possible are obtained as grip position candidates, and then a gripposition is selected from among the grip position candidates. Forexample, the evaluation index calculation portion 8 q calculates a scoreas an evaluation index of each grip position candidate, and selects agrip position candidate having the highest score as a grip position. Aposition of a workpiece located at a high position, in other words, at ahigher position in a bulk workpiece group may be selected as a gripposition on the basis of height information of a workpiece. Preferably,the calculation unit 10 selects a grip position from among a pluralityof grip solutions by taking into consideration both of a score andheight information. In the above-described way, it is possible toperform more appropriate picking.

Second Procedures During Actual Operation

The above description relates to procedures during an actual operationin a state a search model is registered according to the procedures inFIG. 28. When a search model is registered, as illustrated in FIG. 48,the search model may be registered in a state of imposing a restrictionon an attitude thereof. Here, a description will be made of proceduresduring an actual operation of performing a picking operation on anactual bulk workpiece group in a state in which the search model isregistered according to the procedures in FIG. 48, with reference to aflowchart of FIG. 64. First, in step S6401, three-dimensionalmeasurement starts to be performed on bulk workpieces. Herein, thethree-dimensional measurement is performed by imaging a bulk workpiecegroup in the sensor unit, and thus a three-dimensional shape havingheight information is acquired.

Next, in step S6402, a three-dimensional search is performed on theobtained three-dimensional shape of the workpiece group by using aworkpiece model, and a position and an attitude of each workpiece aredetected.

Next, in step S6403, an attitude not included in a range is excludedfrom a set range of an inclined angle and a rotation angle.

Next, in step S6404, with respect to a single detected workpiece, aposition and an attitude at which an end effector is to be disposed arecomputed on the basis of a position of the workpiece and a grip attitudeof the workpiece registered during setting.

Next, in step S6405, interference determination of whether or not theend effector interferes with a peripheral object at the computedposition is performed by using an end effector model.

In step S6406, it is determined whether or not the end effectorinterferes, and, in a case where the end effector does not interfere, itis determined that there is a grip solution for this workpiece, and theprocess is finished.

On the other hand, in a case where it is determined that the endeffector interferes, the flow proceeds to step S6407, and it isdetermined whether or not there are other grip positions registered forthis workpiece. In a case where other grip positions are registered, theflow returns to step S6404, and the processes are repeatedly performedon the grip positions.

On the other hand, in a case where other grip positions are notregistered, the flow proceeds to step S6408, and it is determinedwhether or not there are other detected workpieces. In a case wherethere are other workpieces, the flow returns to step S6404, and theprocesses are repeatedly performed on other workpieces instead of theworkpiece. In a case where there are no other workpieces, it isdetermined that there is no grip solution, and the process is finished.

Interference Determination

Here, a description will be made of an interference determination methodusing an end effector model in step S6304 in FIG. 63 or in step S6405 inFIG. 64 described above. When a workpiece is gripped by an end effector,if the end effector interferes with a peripheral obstacle such asanother workpiece or a storage container, gripping cannot be accuratelyperformed. Therefore, a position or an attitude of the end effector ofwhen an end effector model grips a workpiece model at a grip positioncandidate is calculated by the interference determination portion 8 m inFIG. 31 in advance, and thus an interference determination with aperipheral member is performed. In this case, a three-dimensional pointgroup of a bulk workpiece group or a storage container, obtained throughactual measurement in the sensor unit 2, is used as a peripheral member.A position of a storage container or the like may be registered inadvance, and interference determination with an end effector model maybe performed. On the other hand, three-dimensional CAD data of an endeffector is generally formed of polygon data. For example, STL datawhich is frequently used as three-dimensional CAD data is expressed byan aggregate of minute triangles called a polygon.

In a case where interference determination is performed by using thepolygon data and the three-dimensional point group data, in the relatedart, it is determined whether each three-dimensional point formingthree-dimensional point group data is located inside or outside an endeffector model, and, in a case where the three-dimensional point islocated inside the end effector model, it is determined thatinterference occurs, and, in a case where the three-dimensional point islocated outside the end effector model, it is determined thatinterference does not occur. However, in this method, calculation orcomparison is required to be performed on each point, and, thus, if anamount of data increases, a calculation amount also increases.

Procedures of Interference Determination Using Section Model

Therefore, in each embodiment of the present invention, a section modelis created on the basis of polygon data of an end effector model, eachpoint of three-dimensional point group data is projected onto thesection model, and it is determined whether the point is located insideor outside the section model such that interference therebetween isdetermined. Such interference determination is performed by theinterference determination portion 8 m in FIG. 31. Here, a descriptionwill be made of procedures of performing interference determination withreference to a flowchart of FIG. 65.

First, in step S6501, polygon data of an end effector is read. Next, instep S6502, a section model is created on the basis of the polygon dataof the end effector. The section model is generating by the sectionmodel generation portion 8 s in FIG. 26. Here, a description will bemade of a method of the section model generation portion 8 s creating asection model with reference to FIGS. 66 to 67E. First, a fundamentalaxis BSL is set for polygon data of the end effector model EEMillustrated in FIG. 66. Preferably, the fundamental axis BSL is setalong a longitudinal direction of the end effector model EEM. The endeffector model is cut in an orthogonal plane orthogonal to thefundamental axis BSL such that a plurality of sections are created.Here, a position of creating a section is, for example, a vertexposition of the polygon data. Alternatively, a plurality of sections maybe created along the fundamental axis at a predetermined distanceinterval, and then obtained sections are arranged according to shapes.For example, a section having the same shape is excluded. Thefundamental axis BSL is preferably set such that a total number ofnecessary sections is reduced when the sections are created. Forexample, the section model generation portion may automaticallycalculate a direction of the fundamental axis BSL such that the numberof sections is small. A section position for acquiring a section at theset fundamental axis BSL may be designated such that, for example, asection having a great change is automatically extracted.

In the above-described way, it is possible to create a section modelhaving a shape of each section along the fundamental axis BSL of the endeffector model EEM and a section position on the fundamental axis BSLcorresponding to each section. For example, in the example illustratedin FIG. 66, a section SS1 to a section SS5 respectively illustrated inFIGS. 67A to 67E are obtained at five positions such as sectionpositions SP1 to SP5 along the fundamental axis BSL. It is determinedwhether or not the end effector model EEM interferes with athree-dimensional point group by using the section models obtained inthe above-described way.

Specifically, in step S6503, a three-dimensional point is selected as aninterference determination target from the three-dimensional pointgroup, and a section for interference determination is selected fromamong a plurality of sections of the section models on the basis of aposition of the point in the direction of the fundamental axis BSL. Thesection position set for each section is used for the selection. Forexample, a case is assumed in which interference determination with theend effector model EEM is performed on a three-dimensional point TDPillustrated in FIG. 68A among three-dimensional points. Thethree-dimensional point TDP is located between the section positions SP3and SP4 in the direction along the fundamental axis BSL. Therefore,interference determination with the three-dimensional point TDP isperformed by using the section SS3 indicating the sectional shapebetween SP3 and SP4.

Specifically, in step S6504, a projection point PP3 which is projectedonto an orthogonal plane including the section SS3 from thethree-dimensional point TDP is calculated. In step S6505, interferencedetermination is performed. Herein, if a position of the projectionpoint PP3 of the three-dimensional point is located outside the sectionSS3 as illustrated in FIG. 68B, it is determined that thethree-dimensional point does not interfere. On the other hand, if aposition of the projection point PP3 of the three-dimensional point islocated inside the section SS3 as illustrated in FIG. 68C, it isdetermined that the three-dimensional point interferes. Finally, in stepS6506, it is determined whether or not there is a remainingthree-dimensional point, and, in a case where there is an unprocessedthree-dimensional point, the flow returns to step S6503, and the aboveprocesses are repeatedly performed. If interference determination on allthree-dimensional points is completed, the process is finished.

In the above-described way, it is possible to perform interferencedetermination between a measured three-dimensional point group and anend effector model. The above description relates to interferencedetermination with three-dimensional point group data, but, the presentinvention is not limited to a three-dimensional point as an interferencedetermination target, and, for example, interference determination maybe performed on other objects, for example, a line or a face accordingto the same procedures.

In the above description, a description has been made of an example inwhich CAD data of an end effector is polygon data, but interferencedetermination may be similarly performed in other CAD data formats aslong as a sectional shape of an end effector can be computed in theformats, without being limited to polygon data. In the above example, adescription has been made of a case where a sectional shape is expressedin a two-dimensional plan view, but a method of holding data of asectional shape is not limited to this form, and, for example, data in aformat such as a set of contour lines may be held.

Interference Determination in Additional Model to which AdditionalRegion is Added

In interference determination on an end effector model, the interferencedetermination may be performed by using an additional model in which anadditional region expressed by a solid basic figure is added to the endeffector model. A description will be made of procedures of interferencedetermination using such an additional model with reference to aflowchart of FIG. 69. Herein, a description will be made of an exampleof performing interference determination on an additional model in whichan additional region obtained through a combination of a cuboid and acylinder as basic figures is added to a three-dimensional CAD model.

First, in step S6901, interference determination is performed on aregion of the cuboid of the basic figures forming the additional region.In step S6902, in a case where it is determined that there isinterference as a determination result, the interference determinationprocess is stopped, the presence of interference is output, and theprocess is finished.

On the other hand, in a case where it is determined that there is nointerference, the flow proceeds to step S6903, and interferencedetermination with the cylinder which is another basic figure of thebasic figures forming the additional region is performed. In step S6904,in a case where it is determined that there is interference as adetermination result, the interference determination process is stopped,the presence of interference is output, and the process is finished.

On the other hand, in a case where it is determined that there is nointerference, the flow proceeds to step S6905, and interferencedetermination with three-dimensional CAD data is performed. In stepS6906, in a case where it is determined that there is interference as adetermination result, the interference determination process is stopped,the presence of interference is output, and the process is finished.

On the other hand, in a case where it is determined that there is nointerference, the absence of interference is regarded, and the processis finished.

As mentioned above, interference determination is sequentially performedfor each basic figure or in the three-dimensional CAD data unit, and, ifit is determined that there is interference in any step, theinterference determination is stopped at this time, and the presence ofinterference is output. A description has been made of an example inwhich an additional region is formed of two basic figures such as acuboid and a cylinder, but, even if the number of basic figuresincreases, and the same procedures are applied. In other words,interference determination is sequentially performed on each basicfigure, and, if it is determined that there is interference every time,the process is stopped.

Embodiment 7

Grip Propriety Determination Verification Function

The above description relates to procedures of determining the presenceor absence of a grip solution as a result of interference determinationduring an actual operation. However, according to the present invention,in addition to determination of the presence or absence of a gripsolution, there may be provided a grip propriety determinationverification function of verifying if a grip solution cannot be obtainedfor what reason with respect to a grip position candidate for which thegrip solution cannot be obtained. For example, grip solution candidatesare displayed in a list form, and, as a result of interferencedetermination, a grip position candidate for which it is determined thatthere is a grip solution is displayed OK, and a grip position candidatefor which it is determined that there is no grip solution is displayedNG. In this state, a grip position determined as being NG is selected,and the absence of a grip solution being determined for what reason isdisplayed. Therefore, a user refers to this information, and can examinewhich grip position can be selected as a grip solution, so as to correcta grip position or add a new grip position. Such an example isillustrated in a block diagram of FIG. 70 as a robot system 7000according to Embodiment 7. The robot system 7000 illustrated in FIG. 70includes a robot setting apparatus 700, a display unit 3B, an operationunit 4, a sensor unit 2, a robot controller 6, and a robot RBT. The samemember as in FIG. 6 is given the same reference numeral, and a detaileddescription thereof will be omitted as appropriate.

Display Unit 3B

The display unit 3B includes an image display region 3 b and a gripsolution candidate display region 3 c. The grip solution candidatedisplay region 3 c includes a workpiece grip propriety display region 3d and a workpiece grip impossibility cause display region 3 e.

The image display region 3 b is a member for displaying an end effectormodel virtually expressing a three-dimensional shape of an end effectorand formed of three-dimensional CAD data in a three-dimensional manneron a virtual three-dimensional space.

The grip solution candidate display region 3 c is a member fordisplaying all grip positions set for a certain workpiece among one ormore workpiece search results searched for by the three-dimensionalsearch portion 8 k in a list form.

The workpiece grip propriety display region 3 d is a member fordisplaying a grip propriety determination result at a grip positiondesignated for each workpiece in the three-dimensional pickdetermination portion 8 l.

The workpiece grip impossibility cause display region 3 e is a memberfor displaying a cause of grip impossibility with respect to a gripposition for which grip is determined as being impossible at a gripposition designated for each workpiece in the three-dimensional pickdetermination portion 8 l.

Robot Setting Apparatus 700

The robot setting apparatus 700 includes an input image acquisition unit2 c, a calculation unit 10, a storage unit 9, an input/output interface4 b, a display interface 3 f, and a robot interface 6 b. The calculationunit 10 includes an end effector model registration portion 8 u, aworkpiece model registration portion 8 t, a grip position specifyingportion 8 d, a search model registration portion 8 g, athree-dimensional search portion 8 k, a three-dimensional pickdetermination portion 8 l, and an inclined angle setting portion 8 n.

The grip position specifying portion 8 d includes a workpiece side griplocation designation portion 8 d 1 and an end effector side grip settingportion 8 d 2. The workpiece side grip location designation portion 8 d1 is a member for designating a grip position at which a workpiece modelvirtually expressing a three-dimensional shape of a workpiece and formedof three-dimensional CAD data is gripped by an end effector model in astate in which the end effector model is displayed in the image displayregion 3 b. The end effector side grip setting portion 8 d 2 is a memberfor designating a grip position of gripping a workpiece for the endeffector model displayed in the image display region 3 b.

The search model registration portion 8 g is a member for registering asecond workpiece model virtually expressing a three-dimensional shape ofa workpiece as a search model for a three-dimensional search forspecifying an attitude and a position of each workpiece with respect toa plurality of workpiece groups included in an input image. The secondworkpiece model registered as a search model is preferably a workpiecemodel for which a grip position is designated by the workpiece side griplocation designation portion 8 d 1. Consequently, a search model forperforming a three-dimensional search is commonized to a workpiece modelfor designating a grip position, and thus a user can achievelabor-saving for setting work. Since a workpiece model for searching fora workpiece which can be gripped matches a workpiece model forperforming grip determination during an actual operation, it is examinedwhether or not grip is possible at a grip position set for the searchedworkpiece model, and thus it is possible to perform a process with highefficiency.

The inclined angle setting portion 8 n is a member for setting anallowable inclined angle range with respect to an attitude of aworkpiece.

The three-dimensional pick determination portion 8 l is a member fordetermining whether or not a workpiece can be gripped by an end effectorat a grip position designated for the workpiece by the workpiece sidegrip location designation portion 8 d 1 on the basis of a search resultof each workpiece searched for by the three-dimensional search portion 8k. The three-dimensional pick determination portion 8 l includes aninterference determination portion 8 m and an angle determinationportion 8 o.

The interference determination portion 8 m is a member for determiningthe presence or absence of interference with an object present around aworkpiece at a grip position designated for the workpiece by theworkpiece side grip location designation portion 8 d 1 on the basis of asearch result of each workpiece searched for by the three-dimensionalsearch portion 8 k. The three-dimensional pick determination portion 8 ldetermines that grip is impossible for a workpiece for which it isdetermined that there is interference by the interference determinationportion 8 m. Consequently, a cause of being incapable of gripping aworkpiece is displayed, and this contributes to resetting of a gripposition, for example, since a user easily examines which grip positionis preferably added.

The angle determination portion 8 o is a member for determining whetheror not an attitude of a search result of a workpiece searched for by thethree-dimensional search portion 8 k is within an inclined angle rangeset by the inclined angle setting portion 8 n.

In a case where the angle determination portion 8 o determines that anattitude of a search result of a workpiece searched for by thethree-dimensional search portion 8 k is not within an inclined anglerange set by the inclined angle setting portion 8 n, thethree-dimensional pick determination portion 8 l determines that theworkpiece cannot be gripped. Consequently, in a case where an attitudeof a workpiece is too steep, and thus the workpiece cannot be gripped,or the accuracy of three-dimensional measurement cannot be expected,this is excluded such that wrong selection or wrong determination of agrip position can be prevented, and thus it is possible to increasereliability.

Grip Solution Candidate List Display Function

Next, a description will be made of procedures of displaying the reasonfor a poor grip solution with reference to a flowchart of FIG. 71.Procedures during setting and procedures during an actual operation arefundamentally the same as those in FIGS. 27, 28 and 63 described above,and thus detailed description thereof will be omitted as appropriate.

First, in step S7101, three-dimensional measurement is performed on atarget workpiece group. Herein, actually measured data having shapeinformation is acquired by using the sensor unit with respect to a bulkworkpiece group, and is used as data of an input image.

Next, in step S7102, a three-dimensional search is performed on theinput image, and a position and an attitude of each workpiece in theinput image are detected.

In step S7103, a verification target workpiece is selected from amongdetected workpieces.

In step S7104, a position and an attitude of an end effector model ofwhen the selected workpiece is gripped by an end effector are calculatedon the basis of a detection position of the workpiece and a gripattitude registered for the workpiece in advance.

Next, in step S7105, it is determined whether or not an inclined angleat the calculated position and the attitude of the end effector model isnot included in a set range. Here, in a case where it is determined thatthe inclined angle is not included in the set range, the flow proceedsto step S7106 in which it is determined that grip is poor, the cause ofthe poor grip is set as “inclined angle”, and then the flow jumps tostep S7111.

On the other hand, in a case where it is determined that the inclinedangle of the end effector model is included in the set range in stepS7105, the flow proceeds to step S7107 in which it is determined whetheror not interference determination between the end effector model and aperipheral object at the calculated position is performed. Here, theperipheral object is a storage container or another workpiece presentaround the end effector model. A peripheral object is modeled withthree-dimensional CAD data or the like in advance, and it is determinedwhether or not the peripheral object interferes with the end effectormodel through calculation in a case where the end effector model ismoved to a position and an attitude of gripping a workpiece model.

In step S7108, in a case where it is determined that the end effectormodel interferes with the peripheral object as a result of interferencedetermination, poor grip is determined in step S7109, and a cause of thepoor grip is set as “point group interference”, and then the flow jumpsto step S7111.

On the other hand, in step S7108, in a case where it is determined thatthe end effector model does not interfere with the peripheral object asa result of interference determination, good grip is determined in stepS7110, and then the flow proceeds to step S7111.

In step S7111, it is determined whether or not other grip attitudes areset for the selected workpiece, and, in a case where there are othergrip attitudes, the flow returns to step S7104, and the above-describedprocesses are repeatedly performed. On the other hand, in a case whereit is determined that there is no grip attitude, in step S7112, adetermination result of good grip and poor grip and a cause of the poorgrip are displayed in the workpiece grip impossibility cause displayregion 3 e with respect to all of the grip solution candidates.

In the above-described way, it is verified whether or not a givenworkpiece group can be gripped, and causes of grip impossibility can belisted.

Grip Simulation

Next, a description will be made of details of grip simulation ofperforming specific grip determination. Herein, a description will bemade of an example of performing determination of whether or not grip ispossible in a case where bin picking is performed by using a workpieceWK10 as illustrated in FIG. 72. The fundamental direction imagegeneration portion 8 e′ generates fundamental direction imagescorresponding to six drawings of the workpiece WK10, and the searchmodel registration portion 8 g registers each fundamental directionimage as a search model. FIGS. 73A to 73E illustrate examples of thefundamental direction images. In these figures, FIG. 73A is a heightimage A in which the workpiece in FIG. 72 is viewed from a positivedirection side of a Z axis; FIG. 73B is a height image B in which theworkpiece in FIG. 72 is viewed from a negative direction side of the Zaxis; FIG. 73C is a height image C in which the workpiece in FIG. 72 isviewed from a positive direction side of an X axis; FIG. 73D is a heightimage D in which the workpiece in FIG. 72 is viewed from a negativedirection side of the X axis; FIG. 73E is a height image E in which theworkpiece in FIG. 72 is viewed from a positive direction side of a Yaxis; and FIG. 73F is a height image F in which the workpiece in FIG. 72is viewed from a negative direction side of the Y axis.

Workpiece Selection Screen 210

In order to determine whether or not a workpiece can be gripped by anend effector, a three-dimensional search is performed on an input imageincluding a bulk workpiece group, and a target workpiece is selected ina state in which workpieces are detected (step S7103 in FIG. 71).Selection of a workpiece is performed on, for example, a workpieceselection screen 210 illustrated in FIG. 74. The workpiece selectionscreen 210 illustrated in FIG. 74 includes an image display field 141and an operation field 142. An input image which is actually measureddata obtained by imaging the bulk workpiece group is displayed in theimage display field 141. A search result is displayed to overlap theinput image as a point group at a position where a workpiece isdetected. The screen of the image display field 141 is dragged, and thusa viewpoint can be changed.

Label Number and Model Number

The operation field 142 includes a target workpiece selection field 211for selecting a target workpiece, a detection search model display field212 indicating a search model used for three-dimensionally searching forthe target workpiece, and a “grip check” button 213 for displaying allgrip position candidates for the selected workpiece in a list form. Theexample illustrated in FIG. 74 shows a state in which 18 workpieces arepresent as a result of three-dimensionally searching the input image,and a third workpiece is selected from among the workpieces. Here,identification information is set for a search result in order todifferentiate a detected search result. Herein, a serial label number isset as the identification information. “3” is displayed as a labelnumber of the selected target workpiece in the target workpieceselection field 211 in FIG. 74. A search result of the selected targetworkpiece is displayed in the image display field 141. Specifically, asa scene of a search result, feature points of a corresponding searchmodel are displayed to be superimposed on the point group of the inputimage. If a label number of the target workpiece is changed in thisstate, the workpiece selected in the image display field 141 is alsochanged according thereto. A search model E (the height image in FIG.73E) is displayed as a search model used to three-dimensionally searchfor the target workpiece in the detection search model display field212. Individual identification information is also given to a searchmodel, and is referred to as a model number herein. In FIG. 74, “E” of aworkpiece model having the serial label number “3” is displayed as amodel number in the detection search model display field 212.

Grip Solution Candidate Display Screen 220

Grip positions set for each workpiece, that is, grip solution candidatesmay be displayed in the grip solution candidate display region 3 c in alist form. In the example of the workpiece selection screen 210 in FIG.74, if the “grip check” button 213 is pressed, a grip solution candidatedisplay screen 220 in FIG. 75 is displayed in the display unit. On thegrip solution candidate display screen 220 which is a form of the gripsolution candidate display region 3 c, all grip positions set for atarget workpiece selected in the target workpiece selection field 211 inFIG. 74 are listed in a grip solution candidate display field 221 asgrip solution candidates. The grip solution candidate display field 221includes a workpiece grip position display field 223 displaying a labelnumber of a grip position, a workpiece grip propriety display field 224displaying a workpiece grip propriety determination result, and aworkpiece grip impossibility cause display field 225 displaying a causeof grip impossibility for each grip solution. In the example illustratedin FIG. 75, all of five grip candidates are listed with respect to thethird search result target workpiece selected in the target workpieceselection field 211, and determination results of good grip (OK) andpoor grip (NG) and causes of the poor grip are displayed. The workpieceselected in the image display field 141 and an end effector modelgripping the workpiece are displayed at a grip attitude at the gripposition, so as to correspond to the grip position of the workpieceselected in the grip solution candidate display field 221. If selectionof the grip position is changed in the grip solution candidate displayfield 221, the workpiece in the image display field 141 is also changedaccording thereto, and an attitude of the end effector model grippingthe workpiece is also updated.

The example illustrated in FIG. 75 shows a state in which the workpiecein FIG. 73E is gripped at the grip position candidate having the labelnumber “3”. Herein, since a determination result is good grip, nothingis displayed in the workpiece grip impossibility cause display field225. On the other hand, if a grip position candidate having the labelnumber “4” is selected in the workpiece grip impossibility cause displayfield 225, the display in the image display field 141 is also changed asillustrated in FIG. 76. Since the grip position candidate having thelabel number “4” has a determination result of poor grip, a cause to bedetermined as being grip impossibility is displayed in the workpiecegrip impossibility cause display field 225. An end effector modelcorresponding thereto is displayed in the image display field 141, andthus it is possible to visually check an actual state of a gripattitude. Herein, “point group interference” is displayed, and it can beseen that an end effector interferes with a flat surface of a storagecontainer. As a result, a user can examine adding of a grip position ata position or an attitude not causing interference with respect to theworkpiece having such an attitude.

Similarly, if a grip position candidate having the label number “2” isselected in the workpiece grip impossibility cause display field 225,the grip solution candidate display screen 220 illustrated in FIG. 77 isdisplayed, and an “inclined angle” is displayed in the workpiece gripimpossibility cause display field 225 as a cause of the grip positioncandidate having the label number “2” being determined as being poorgrip. An end effector model corresponding thereto is displayed in theimage display field 141, an end effector is in a state of trying to gripa workpiece at a steep inclined angle, and thus it can be seen that theend effector model is excluded from an interference determination targetin the first place.

A display aspect of an end effector displayed in the image display field141 may be changed according to a determination result. In the exampleillustrated in FIG. 75, the end effector model is displayed white in acase of good grip, and, in the example illustrated in FIG. 76 or 77, theend effector model is displayed red in a case of poor grip.Consequently, a user can easily visually differentiate gripdetermination results from each other. When a grip position is added,display in the image display field 141 may be updated such that adisplay aspect is changed at a time point at which interference does notoccur by adjusting an attitude of an end effector model. With thisconfiguration, a user can change, for example, an attitude of the endeffector model until a color of the end effector model is changed towhile from red, and thus it is possible to obtain an advantage thatattitude adjustment work can be intelligibly performed.

Since the grip propriety determination verification function is providedas mentioned above, in a case where grip impossibility is determined fora workpiece desired to be gripped, the function is a guideline forexamining work such as adding a new grip position or changing setting ofan existing grip position. For example, in the examples illustrated inFIGS. 78 and 79, two grip position candidates are present for theworkpiece of the search model C (FIG. 73C), but neither thereof aregripped due to point group interference. In both cases, it can be seenthat this is because the tip of the claw of the end effector modelinterferes with another workpiece as indicated by a dashed circle in theimage display field 141. Thus, if a new grip position is added such thata location on an opposite side to the current grip position is gripped,it can be seen that a good grip solution is obtained. Therefore, ifsetting is performed with the grip position specifying portion 8 d suchthat a new grip position (grip attitude C-002) is added such that alocation on an opposite side to the current grip position is gripped, adetermination result of good grip is obtained as illustrated in FIG. 80.In the above-described way, when a user performs adjustment work of agrip position or a grip attitude, an environment in which acountermeasure is easily examined is provided, and thus a robot systemin which setting is easily performed is realized.

A threshold value of an inclined angle of an end effector may be changedthrough setting performed by a user. For example, in a case where a boxhaving a deep bottom is used a storage container for workpieces, if aninclination of an end effector increases, not only the end effector butalso the arm portion of the robot easily collides with a wall of thestorage container, and thus an angle range is set to be narrow.Conversely, in a case where a box having a shallow bottom is used as astorage container, if an end effector does not interfere, the armportion of the robot scarcely collides with a wall of the storagecontainer, and thus an angle range is set to be wide. As mentionedabove, if setting of an angle range is adjusted, it is possible toflexibly adjust grip propriety determination according to an actualsituation.

In the above-described way, in a case where all grip solutions for aselected workpiece are displayed in a list form and are determination asbeing poor grip, causes of the poor grip are also displayed. Therefore,in a case where there is a workpiece which seems to be gripped but isnot a grip solution candidate for a long period of time, it becomeseasier to specify a cause thereof. Since the cause can be specified, itbecomes easier to understand what kind of new grip attitude ispreferably added.

Embodiment 8

In the above-described example, a description has been made of theexample in which a search result is individually acquired for eachsearch model during a three-dimensional search. In other words, athree-dimensional search is performed on a plurality of fundamentaldirection images indicating different faces of the same workpiece, andobtained results are recognized as different workpieces. In other words,as a result of the different faces of the same workpiece beingindividually searched, the workpiece may be detected as differentworkpieces. On the other hand, in a three-dimensional search of therelated art, a three-dimensional shape indicating a single workpiecemodel is searched, and thus such a case does not occur, and differentfaces of the same workpiece are caused to be detected as a singleworkpiece. However, in a workpiece having many faces, a search iscomplicated, and thus a probability of wrong detection increases. Incontrast, in the method according to the above-described embodiment,since a simple face is searched, it is possible to simplify a searchprocess, and the method is advantageous in terms of achievement of lowload, high speed, and the like. On the contrary, as a result of eachface being searched, as described above, obtained search results may berecognized as separate workpieces, and thus there is a problem in thatdifferent faces of even the same workpiece are individually detected.

Therefore, obtained search results are integrated with each other so asto be aggregated for each workpiece, or a search result of another faceis estimated from a search result of a certain face. Consequently, aface which cannot be detected in a three-dimensional search or a facehaving low detection accuracy can be used as a grip solution candidate.Such an example is illustrated in a block diagram of FIG. 81 as a robotsystem according to Embodiment 8. The robot system 8000 illustrated inFIG. 81 includes a robot setting apparatus 800, a display unit 3B, anoperation unit 4, a sensor unit 2, and a robot RBT. The same member asin FIG. 6 and FIG. 70 is given the same reference numeral, and adetailed description thereof will be omitted as appropriate.

Robot Setting Apparatus 800

The robot setting apparatus 800 includes an input image acquisition unit2 c, a calculation unit 10, a storage unit 9, an input/output interface4 b, a display interface 3 f, and a robot interface 6 b. The calculationunit 10 includes a fundamental direction image generation portion 8 e′,a grip position specifying portion 8 d, a search model registrationportion 8 g, a three-dimensional search portion 8 k, an image estimationportion 8 z, a search result integration portion 8 p, athree-dimensional pick determination portion 8 l, and an inclined anglesetting portion 8 n.

The fundamental direction image generation portion 8 e′ is a member forgenerating a plurality of height images in which a workpiece model isviewed from respective axis directions of three axes orthogonal to eachother on a virtual three-dimensional space as fundamental directionimages.

The grip position specifying portion 8 d is a member for specifying aplurality of grip positions at which a workpiece model indicated by oneof the fundamental direction images generated by the fundamentaldirection image generation portion 8 e′ is gripped by an end effector.The grip position specifying portion 8 d includes a workpiece side griplocation designation portion 8 d 1 and an end effector side grip settingportion 8 d 2.

The workpiece side grip location designation portion 8 d 1 is a memberfor designating a grip position at which a workpiece model virtuallyexpressing a three-dimensional shape of a workpiece and formed ofthree-dimensional CAD data is gripped by an end effector model in astate in which the end effector model is displayed in the image displayregion 3 b.

The end effector side grip setting portion 8 d 2 is a member fordesignating a grip position of gripping a workpiece for the end effectormodel displayed in the image display region 3 b.

The search model registration portion 8 g is a member for registering asearch model used to perform a three-dimensional search for specifyingan attitude and a position of each workpiece from a plurality offundamental direction images generated by the fundamental directionimage generation portion 8 e′ with respect to a plurality of workpiecegroups included in an input image acquired by the input imageacquisition unit 2 c. The search model registration portion 8 g mayregister a relative position between a plurality of registered searchmodels as relationship information (details thereof will be describedlater).

The image estimation portion 8 z is a member for estimating, by using anestimation image, a position and an attitude of a non-searchedfundamental direction image not included in a search result in thethree-dimensional search portion for a workpiece model indicated by thesearch result, on the basis of a relative positional relationship with afundamental direction image of another search model registered for theworkpiece model, each search model used for a search being a fundamentaldirection image in which an original workpiece model is viewed from acertain direction, with respect to each search result which is searchedfor by the three-dimensional search portion 8 k and is extracted from aninput image in the search model unit. For example, in a case where it isdetermined by the angle determination portion 8 o which will bedescribed later that an attitude of a workpiece search result searchedfor by the three-dimensional search portion 8 k is included in aninclined angle range set by the inclined angle setting portion 8 n, theimage estimation portion 8 z may estimate an estimation image having arelative positional relationship with the search result.

The search result integration portion 8 p is a member for integratingadjacent search results among a plurality of search results as anintegration result regarding a common workpiece, on the basis of arelative positional relationship of each search model used for a searchbeing a fundamental direction image in which an original workpiece modelis viewed from a certain direction, with respect to each search resultwhich is searched for by the three-dimensional search portion 8 k and isextracted from an input image in the search model unit.

The three-dimensional pick determination portion 8 l is a member fordetermining whether or not a workpiece model can be gripped by an endeffector at a grip position designated for the workpiece model in thegrip position specifying portion 8 d, on the basis of an integrationresult obtained through integration in the search result integrationportion 8 p, and each search result not integrated, searched in thethree-dimensional search portion 8 k. The three-dimensional pickdetermination portion 8 l includes an interference determination portion8 m and an angle determination portion 8 o.

Consequently, since a position and an attitude of a workpiece can bemore accurately estimated by using relationship information betweenfaces without individually performing grip determination on a searchresult obtained through a three-dimensional search, a face not searchedfor or a face having low accuracy of a search result can be detected,even a face which is normally difficult to search for can be examined asa grip position candidate, and thus it is possible to increase apossibility that a grip solution can be obtained.

Procedures of Registering Search Model Including RelationshipInformation Between Faces

Procedures of performing setting such as registration of a workpiecemodel or an end effector model or registration of a grip position on therobot setting apparatus may use, for example, the procedures shown inthe flowchart of FIG. 27. Here, with reference to a flowchart of FIG.82, a description will be made of procedures of registering a searchmodel including relationship information between faces of a workpiece byusing three-dimensional CAD data as a search model in step S2701 in FIG.27.

First, in step S8201, a three-dimensional CAD data model of theworkpiece is read.

Next, in step S8202, the center of a circumscribing cuboid of thethree-dimensional CAD data model is corrected to the origin of thethree-dimensional CAD data.

In step S8203, height images viewed from respective directions of “top”,“bottom”, “left”, “right”, “front”, and “rear” are generated asfundamental direction images. The fundamental direction images aregenerated by the fundamental direction image generation portion 8 e′ inFIG. 81. Here, in a case where a height image is generated on the basisof the three-dimensional CAD data, the height image is generated suchthat the origin of CAD is the center of the height image.

Next, in step S8204, a height image having the same viewing way isdeleted from the generated height images.

In step S8205, a relative position among a plurality of registeredsearch models is registered as relationship information. Herein, thesearch model registration portion 8 g stores relationship information ofthe remaining height images and the respective faces such as top,bottom, left, right, front and rear faces.

Finally, in step S8205, a search model is registered by using thegenerated height images. In the above-described way, the search modelincluding the relationship information among the faces of the workpieceis registered.

Relationship Information of Faces

Here, a description will be made of relationship information indicatinga relative positional relationship between faces in which a workpiece isviewed from specific directions. For example, in a case of the workpieceas illustrated in FIG. 7, faces having different viewing ways are fourfaces illustrated in FIGS. 9A to 9D. In these figures, the image of themodel A illustrated in FIG. 9A is a height image viewed from thepositive direction of the X axis. However, the image is the same as animage viewed from the negative direction of the X axis. The image of themodel B illustrated in FIG. 9B is a height image viewed from thepositive direction of the Y axis, and is also the same as an imageviewed from the negative direction of the Y axis. On the other hand, theimage of the model C illustrated in FIG. 9C is the same as only an imageviewed from the positive direction of the Z axis. The image of the modelD illustrated in FIG. 9D is the same as only an image viewed from thenegative direction of the Z axis.

Here, information indicating that each of the images of the models A, B,C and D as height images generated in advance is the same as an image inwhich a three-dimensional CAD data model of the workpiece illustrated inFIG. 7 is viewed from a predetermined coordinate axis direction at apredetermined rotation state is assumed to be information indicatingrelationship information of faces. If there is relationship informationof the faces, an attitude of original three-dimensional CAD data isobtained on the basis of a search result of each model. For example, themodel B in FIG. 9B is present to be adjacent to each of the left andright faces of the model A in FIG. 9A. The model C in FIG. 9C is presentto be adjacent to the top face of the model A, and the model D in FIG.9D is present to be adjacent to the bottom face thereof. As a result, ina case where detection results of the model A and the model B areadjacent to each other in searched images obtained as a result of athree-dimensional search, it can be seen that the results are obtainedby searching the same workpiece. Therefore, the search results can beintegrated into a single search result. Here, a set of integrated searchresults is referred to as an integration result.

An evaluation index may be updated by integrating search results witheach other. In other words, in a case where even a search result havinglow accuracy may be obtained as a search result having high accuracy inan integration result, an evaluation index of the integration result maybe heightened, and thus the search result may be prioritized as anaccurate grip solution when a grip position is selected.

Non-Searched Fundamental Direction Image

Even if there is a face not searched as in a case where athree-dimensional search fails or a search is not performed, estimationusing an integration result can be performed. In other words, when anintegration result of a workpiece is obtained by integrating a pluralityof search results with each other, even in a case where there is a facenot detected through a three-dimensional search among faces forming theworkpiece, if search results can be obtained from other faces, anattitude or a position of the workpiece can be calculated. As a result,in a case where a workpiece model in which a grip position is registeredfor a workpiece in advance or a search model for a three-dimensionalsearch has been registered, information regarding a face which is notsearched for can be estimated on the basis of an attitude of theworkpiece. Therefore, a face (also referred to as a non-searchedfundamental direction image) which is not obtained as a result of thesearch and is obtained through estimation is also used as a gripposition candidate, and thus a grip position of the face which is notactually searched for can be examined as a grip solution candidate, sothat it is possible to obtain an advantage that an appropriate gripsolution can be easily obtained. For example, in a case where the modelB illustrated in FIG. 9B is obtained as a result of a three-dimensionalsearch, it can be estimated that the face of the model A illustrated inFIG. 9A is present on the side thereof. Consequently, even if the modelA is not actually detected as a search result, the face of the model Acan be estimated on the basis of the search result of the model B, agrip attitude registered for the face of the model A can be used as agrip solution candidate, and thus the number of grip solution candidatesis increased, so that picking is easily performed.

Calculation of Grip Solution Using Relationship Information of Faces

Next, a description will be made of an effect of grip solutioncalculation using relationship information of faces by using theworkpiece WK10 illustrated in FIG. 72. The fundamental direction imagesillustrated in FIGS. 73A to 73F are generated by the fundamentaldirection image generation portion 8 e′ on the basis ofthree-dimensional CAD data of the workpiece WK10, and are registered assearch models by the search model registration portion 8 g. In thesefigures, the image illustrated in FIG. 73A is a height image viewed fromthe positive direction of the Z axis, and is used as an image of themodel A. Similarly, the image illustrated in FIG. 73B is a height imageviewed from the negative direction of the Z axis, and is used as animage of the model B. The image illustrated in FIG. 73C is a heightimage viewed from the positive direction of the X axis, and is used asan image of the model C. The image illustrated in FIG. 73D is a heightimage viewed from the negative direction of the X axis, and is used asan image of the model D, the image illustrated in FIG. 73E is a heightimage viewed from the positive direction of the Y axis, and is used asan image of the model E, and the image illustrated in FIG. 73F is aheight image viewed from the negative direction of the Y axis, and isused as an image of the model F.

Here, it is assumed that grip attitudes at which the workpiece modelWM10 is gripped by an end effector model EM10 are respectivelyregistered for the fundamental direction images by the grip positionspecifying portion 8 d as illustrated in FIGS. 83A to 83F. Here, FIG.83A illustrates a state in which a grip attitude A-000 is registered forthe fundamental direction image in FIG. 73A. FIG. 83B illustrates astate in which a grip attitude B-000 is registered for the fundamentaldirection image in FIG. 73B. FIG. 83C illustrates a state in which agrip attitude C-000 is registered for the fundamental direction image inFIG. 73C, FIG. 83D illustrates a state in which a grip attitude D-000 isregistered for the fundamental direction image in FIG. 73D, FIG. 83Eillustrates a state in which a grip attitude E-000 is registered for thefundamental direction image in FIG. 73E, and FIG. 83F illustrates astate in which a grip attitude F-000 is registered for the fundamentaldirection image in FIG. 73F. In this case, a description will be made ofcalculation of a grip solution using relationship information of faceswith reference to FIGS. 84, 85 and 86.

The workpiece selection screen 210 in FIG. 84 displays a state in whichtwelve workpieces are detected as a result of performing athree-dimensional search on an input image obtained by imaging a bulkworkpiece group. The input image is displayed in the image display field141. The operation field 142 includes the target workpiece selectionfield 211 for selecting a target workpiece, the detection search modeldisplay field 212, and the “grip check” button 213 for displaying gripposition candidates for the selected workpiece in a list form. In theexample illustrated in FIG. 84, among the twelve workpieces obtainedthrough the three-dimensional search, the second workpiece is selectedin the target workpiece selection field 211, and a search result of theselected second target workpiece is displayed in the image display field141. Specifically, as a scene of the search result, feature points ofthe corresponding search model are displayed to overlap a point group ofthe input image. The search models C, F and A are displayed as searchmodels used to detect the second target workpiece in the detectionsearch model display field 212. In this state, the three search modelsC, F and A (FIGS. 73C, 73F and 73A) are integrated with each other.

Here, if the relationship information of faces of a workpiece is notused, the model F or the model A may be hardly detected as a searchresult since only a part of the side face is viewed on the input imageas displayed in the image display field 141. Even if the model isdetected, an evaluation index may be low, and, as a result, the priorityis low. In contrast, by using the relationship information, as in FIG.84, not only the search model C but also the search model F or thesearch model A can be detected or estimated, and can thus be used as agrip position candidate.

If the “grip check” button 213 is pressed on the workpiece selectionscreen 210 in FIG. 84, the grip solution candidate display screen 220 inFIG. 85 is displayed. On the grip solution candidate display screen 220,grip positions set for the second target workpiece selected in thetarget workpiece selection field 211 in FIG. 84 are listed in the gripsolution candidate display field 221 as grip solution candidates. In thegrip solution candidate display field 221, workpiece grip proprietydetermination results for grip positions (FIGS. 83C, 83F and 83A)respectively set for the search models C, F and A displayed in thedetection search model display field 212 are displayed in the workpiecegrip propriety display field 224, and causes of grip impossibility aredisplayed in the workpiece grip impossibility cause display field 225.First, for the search model C, it is displayed that a determinationresult is poor grip, and a cause thereof is interference with pointgroup data. The image display region displays a state in which the endeffector model EM10 trying to grip the workpiece at an attitudecorresponding to the grip position C-000 is displayed red, and cannotgrip the workpiece due to interference. Here, if the relationshipinformation of faces is not used, only the search model C isthree-dimensionally searched for, and, as a result, a grip solutioncandidate is only C-000 of the search model C, and thus a grip solutioncannot be obtained.

In contrast, by using the relationship information, as illustrated inFIGS. 85 and 86, it can be seen that the search models F and A are alsotargets in addition to the search model C, and F-000 and A-000 are addedas grip position candidates. Of the added grip position candidates, adetermination result of poor grip is obtained in the grip positioncandidate F-000 of the search model F, but a determination result ofgood grip is obtained in the grip position candidate A-000 of the searchmodel A. As illustrated in FIG. 86, the end effector is displayed white,and a grip solution is obtained with safety without interference withthe point group data. As a result, even in a case where a grip solutionis not obtained unless the relationship information is used, a gripsolution can be obtained by using the relationship information. In otherwords, it is possible to increase a possibility that a grip solution canbe obtained without changing setting of a three-dimensional search, forexample, without adjusting arrangement of a light or a camera orchanging an algorithm or the like of a search, under the same searchcondition, in other words, without changing the accuracy of athree-dimensional search. As mentioned above, by using the relationshipinformation of faces, the number of grip solution candidates can beincreased, and thus a result which is easy to pick can be obtained.

Third Procedures During Actual Operation

Here, a description will be made of procedures of performing athree-dimensional search and grip propriety determination during anactual operation with reference to a flowchart of FIG. 87 in a state inwhich the search model is registered according to the procedures in FIG.82.

First, in step S8701, three-dimensional measurement starts to beperformed on bulk workpieces. Herein, the three-dimensional measurementis performed by imaging a bulk workpiece group in the sensor unit 2illustrated in FIG. 81, and thus an input image including athree-dimensional shape having height information is acquired by theinput image acquisition unit 2 c.

Next, in step S8702, a three-dimensional search is performed on theobtained three-dimensional shape of the workpiece group by using aworkpiece model, and a position and an attitude of each workpiece aredetected. The three-dimensional search is performed by thethree-dimensional search portion 8 k in FIG. 81.

Generation of Integration Result

In step S8703, results indicating the same workpiece are integrated witheach other on the basis of search results of the three-dimensionalsearch and the relationship information of faces. Herein, the searchresult integration portion 8 p correlates search results obtained byimaging the same workpiece with each other by using relationshipinformation registered for a search model used for a search. In otherwords, the search result integration portion 8 p integrates adjacentsearch results with each other among the plurality of search results onthe basis of a relative positional relationship related to a fundamentaldirection image in which a common workpiece model providing searchresults is viewed from a predetermined direction. The integration resultobtained in the above-described way indicates the same workpiece and ishandled in common.

Generation of Non-Searched Fundamental Direction Image

In step S8704, a face which is not detected is estimated on the basis ofthe three-dimensional search results and the relationship information offaces. Herein, a position and an attitude of a non-searched fundamentaldirection image not included in a three-dimensional search resultregarding a workpiece are estimated on the basis of an attitude and aposition of the workpiece indicated by the integration result. Grippropriety determination is performed on a grip position which is set forthe non-searched fundamental direction image by the grip positionspecifying portion 8 d. Consequently, a face of the workpiece which isnot initially searched for as a three-dimensional search result isestimated on the basis of an attitude and a position of the workpieceindicated by an integrated search result, and can thus be set as a grippropriety determination target, and thus it is possible to furtherincrease a possibility that a grip solution can be obtained.

Next, in step S8705, with respect to a single detected workpiece, aposition and an attitude at which an end effector is to be disposed arecomputed on the basis of a position of the workpiece and a grip attitudeof the workpiece registered during setting.

Next, in step S8706, interference determination of whether or not theend effector interferes with a peripheral object at the computedposition is performed by using an end effector model.

In step S8707, it is determined whether or not the end effectorinterferes, and, in a case where the end effector does not interfere, itis determined that there is a grip solution for this workpiece, and theprocess is finished.

On the other hand, in a case where it is determined that the endeffector interferes, the flow proceeds to step S8708, and it isdetermined whether or not there are other grip positions registered forthis workpiece. In a case where other grip positions are registered, theflow returns to step S8705, and the processes are repeatedly performedon the grip positions.

On the other hand, in a case where other grip positions are notregistered, the flow proceeds to step S8709, and it is determinedwhether or not there are other detected workpieces. In a case wherethere are other workpieces, the flow returns to step S8705, and theprocesses are repeatedly performed on other workpieces instead of theworkpiece. In a case where there are no other workpieces, it isdetermined that there is no grip solution, and the process is finished.

In the above-described way, the presence or absence of a grip solutionin which a workpiece can be gripped is determination by using anintegration result or a non-searched fundamental direction image. In acase where a grip solution is obtained, an instruction is given to therobot controller 6 such that the workpiece is gripped by an end effectorat a determined grip position.

An image in the present specification is not limited to strictlyconsecutive data, and indicates a set of discrete data such as a set ofpoint group data.

Embodiment 9

Integration of Evaluation Index

In addition to the procedures, an evaluation index for defining prioritymay be integrated with an integration result. Such an example isillustrated in FIG. 88 as a robot system 9000 according to Embodiment 9.The robot setting apparatus 900 illustrated in FIG. 88 includes anevaluation index calculation portion 8 q and a grip prioritydetermination portion 8 r. The same members as the members illustratedin FIG. 81 or the like are given the same reference numerals, anddetailed description thereof will be omitted.

The evaluation index calculation portion 8 q is a member for calculatingan evaluation index for each of grip position candidates regarding grippositions at which it is determined that a workpiece can be gripped bythe three-dimensional pick determination portion 8 l.

The grip priority determination portion 8 r is a member for determiningpriority in which a workpiece is gripped by an end effector on the basisof the evaluation index calculated by the evaluation index calculationportion 8 q.

The robot setting apparatus calculates an evaluation index for eachsearch result by using the evaluation index calculation portion 8 q. Asthe evaluation index, as described above, a proportion of the number offeature points corresponding to an error of a predetermined distance orless, included in a search result, may be used.

Here, the search result integration portion 8 p adds the highestevaluation index among evaluation indexes calculated for the respectivesearch results forming the integration result by the evaluation indexcalculation portion 8 q, to the integration result. Generally, anevaluation index of a three-dimensional search depends on a searchcondition or an attitude of a workpiece (for example, a case where aninclined angle of the workpiece is large, and sufficient reflected lightcannot be obtained, or, conversely, a case where reflected light isstrong due to a glossy workpiece). Thus, an evaluation index for asearch result may be evaluated to be low such that low priority isobtained, or a workpiece may be excluded from a three-dimensional searchtarget for the reason why reflected light cannot be inherently obtained.In this case, a grip position is a grip position candidate with lowpriority or does not become a grip position candidate, but there is acase where such a workpiece is not disposed at an attitude appropriatefor a three-dimensional search but is disposed at an attitudeappropriate for gripping. Even in this case, in the related art, sincethe workpiece is excluded in the stage of a three-dimensional search, orhas low priority, a grip position has a low probability of beingselected as a grip solution, and cannot be sufficiently used. Incontrast, according to the present embodiment, in a case where anevaluation index for a grip position is low as a three-dimensionalsearch result, but a high evaluation value is obtained as another searchresult adjacent thereto, the grip position can be used as a gripposition candidate with high priority by using this fact. Consequently,even a grip position candidate which is not used in the related art canbe used, and thus it is possible to increase a possibility that a gripsolution can be obtained. In other words, an appropriate workpiece canbe gripped regardless of the accuracy of a three-dimensional search.

In the above-described method, a description has been made of an aspectin which the entire integration result is evaluated by using the highestevaluation index among evaluation indexes for respective search resultsforming an integration result, or an evaluation index for each searchresult is rewritten from an original evaluation value to a higherevaluation value. This is based on the fact that the reliability of athree-dimensional search result is high for a search result for which ahigh evaluation index is obtained. However, the present invention is notlimited to this aspect, and may be applied to another aspect, forexample, evaluation indexes for respective search results forming anintegration result may be averaged, and the average search result may behandled as an evaluation index for the integration result. In any case,according to the present embodiment, a search result having an initiallylow evaluation index is evaluated with an evaluation index higher thanthe initial evaluation index, so as to become a candidate of a gripposition or the like.

GUI of Robot Setting Program

Here, a description will be made of examples of GUIs of a robot settingprogram with reference to FIGS. 89 to 149. FIG. 89 illustrates afunction selection screen 340 for selecting various functions, andavailable functions are listed and displayed in a button form in afunction list display field 341. If a “3D search” button 342 is pressedin this state, a three-dimensional search function is executed.

Three-Dimensional Search Screen

Necessary setting is required to be performed by executing thethree-dimensional search function. Specifically, such setting mayinclude registration of a search model, a registration of a floorsurface or a storage container on or in which a workpiece is placed,setting of a three-dimensional search condition, and the like. Regardingsuch setting, all items may be arbitrarily set by a user, and each itemmay be set while a necessary procedure is presented to the user. Aguidance function of such setting will be described with reference tothree-dimensional search screens in FIGS. 90 to 101.

In each of the three-dimensional search screens, a flow display portion351 in which a flow of procedures to be set is displayed is provided onan upper part of the image display field 141 located on the lower leftside in the screen. In the flow display portion 351, a summary of eachprocedure is indicated in text or a picture, and a procedure which iscurrently set is highlighted. In an example illustrated in FIG. 90, a“model registration” icon 352 is displayed in a bright color, and aremaining icon such as a “floor/box setting” icon 353 or a “searchsetting” icon 354 is displayed in a dark color, and thus a user can benotified of a target which is currently set. Explanation of an itemwhich is currently set is displayed in text or the like in the operationfield 142 provided on the right side in the three-dimensional searchscreen, and setting to be performed can be explained to a user in text.Explanation may be performed not only by using text information but alsothrough a combination with a picture or a moving image as appropriate asnecessary. A setting field or the like for a parameter to be set is alsoprovided.

Registration of Search Model Based on Actually Measured Data

The three-dimensional search screens in FIGS. 90 to 101 show an examplein which actually measured data obtained by imaging a real workpiece inthe sensor unit is registered as a search model. Such registration maybe performed according to the procedures described in the flowchart ofFIG. 27. On the three-dimensional search screens in FIGS. 90 to 101, the“model registration” icon 352 is highlighted in the flow display portion351 as described above as a first procedure.

FIG. 90 illustrates a search model registration method selection screen350 for selecting a method of registering a search model. A modelregistration method selection field 355 for selecting a method ofregistering a model is provided in the operation field 142 of the searchmodel registration method selection screen 350. In the modelregistration method selection field 355, as a model registration method,whether a model obtained by imaging a real workpiece is registered or isregistered from CAD data may be selected with a radio button. In theexample of the search model registration method selection screen 350 inFIG. 90, the method of registering a model obtained by imaging a realworkpiece is selected, and, according thereto, an explanation ofprocedures of registering a model obtained by imaging a real workpieceis displayed in text and pictures in the operation field 142. In thisexample, work to be performed is presented to a user by using textinformation that “three-dimensional information of a workpiece surfaceviewed from the camera is modeled; the number of faces desired to besearched for is registered; and this tool recognizes a position and anattitude of a workpiece by matching the registered surface model withthe workpiece taking various attitudes”, and pictures. Consequently, theuser can easily recognize the work to be performed on this screen. Ifregistration of a real workpiece is selected on the search modelregistration method selection screen 350 in FIG. 90, and then an “OK”button 356 is pressed, imaging of the workpiece is performed.

Actually Measured Data Imaging Unit

Procedures of actually registering a model by using a real workpiece areas in FIGS. 91 to 93. In this example, the model registration work isdivided into three screens, and is sequentially explained to the user.First, the workpiece is imaged as first work of model registration on anactually measured data imaging screen 360 in FIG. 91. In this example,the explanation that “1. dispose the workpiece on the flat surface suchthat a workpiece surface desired to be registered is viewed from thecamera, and press the “measurement execution” button on the lower right;and 2. dispose the model region to surround the workpiece on the leftcamera screen” is displayed in the operation field 142 as work to beperformed by the user. A detailed explanation of optimal arrangement ofa workpiece may be displayed on a separate screen by pressing a “?”button 364. Consequently, it is possible to avoid a situation in which ascreen is hard to view or a user is confused by displaying a lot ofinformation on the screen. In the above-described way, the user disposesthe workpiece at a position where the workpiece can be imaged by thesensor unit, and performs positioning while checking a scene of theworkpiece in the image display field 141. If the “measurement execution”button 361 is pressed, the imaged workpiece is displayed as illustratedin FIG. 91. Herein, images having different colors depending on obtainedheights of the workpiece are displayed as three-dimensional images. Aheight image is not limited to color display, and may express a heightwith a luminance value of a pixel. In the example illustrated in FIG.91, six workpieces having the same shape are arranged, and are displayedat different attitudes. In this state, the user sets a model region 362to surround an image of the workpiece desired to be registered as asearch model. If the model region 362 is set, a “next” button 363 ispressed. Consequently, the screen is changed to a search model exclusionregion setting screen 370 illustrated in FIG. 92.

Background Removal Setting Unit

An exclusion region which is excluded when a search model is registeredis set as second work of the model registration on the search modelexclusion region setting screen 370 in FIG. 92. As an example of afunction using a search model exclusion region, there may be a functionof removing a background such as a floor surface on which a workpiece isplaced when a real workpiece is registered. On the search modelexclusion region setting screen 370 in FIG. 92, the explanation that “1.set background removal such that only the workpiece surface desired tobe registered remains” is displayed in the operation field 142 as thesecond work of model registration. A detailed explanation of abackground removal setting method may be displayed on a separate screenby pressing a “?” button 371. A background removal setting field 372 isprovided on an intermediate part of the operation field 142, and aheight of the background to be removed is designated in a numericalvalue. In this example, 1.5 mm is designated, and a range of the heightof 0 to 1.5 mm in the range designated for the model region 362 isexcluded from a search model registration target. The model region 362displayed in the image display field 141 is updated according to thesetting in the background removal setting field 372, and the regionexcluded from the search model registration target is displayed black.The user can judge whether or not setting of the exclusion region isproper while checking this scene, and may perform resetting asnecessary.

Mask Region Setting Unit

In addition to a height direction, a range desired to be excluded fromthe search model registration target in a plane direction may bedesignated as a mask region. The explanation that “2. dispose a maskregion in a case where there are things other than the workpiece surfacedesired to be registered” is displayed on the lower part in thebackground removal setting field 372 of the search model exclusionregion setting screen 370 in FIG. 92. Similarly, a detailed explanationof a mask region setting method may be displayed on a separate screen bypressing a “?” button 373. If a “mask region” button 374 is pressed, amask region can be set as a region excluded from the search modelregistration target within the range set for the model region 362. Themask region may be set to a predefined shape such as a rectangular shapeor a circular shape, and may also be set through automatic detection ofa free curve or a boundary portion. The model region 362 displayed inthe image display field 141 is updated as described above according tosetting of the mask region.

In the example illustrated in FIG. 92, background removal is set, andthen a mask region is set, but orders thereof may be replaced with eachother. If the region desired to be excluded from the search modelregistration target is set in the above-described way, a “next” button375 is pressed. Consequently, the screen is changed to a rotationsymmetry setting screen 380 illustrated in FIG. 93.

Rotation Symmetry Setting Unit

Rotation symmetry is set as third work of the model registration on therotation symmetry setting screen 380 in FIG. 93. In this example, theexplanation that “1. in a case where workpiece surfaces to be registeredhave rotation symmetry, designate a corresponding workpiece surface” isdisplayed in the operation field 142. Similarly, a detailed explanationof a rotation symmetry setting method may be displayed on a separatescreen by pressing a “?” button 381. A rotation symmetry setting field382 is provided on an intermediate part of the operation field 142, anda rotation symmetry candidate may be selected from options such ascircle symmetry, N-fold symmetry, and none. In a case where the N-foldsymmetry is selected, a numerical value input box for the symmetrynumber is displayed, and the rotation symmetry number corresponding to ashape of the workpiece surface is set. For example, in a case where theworkpiece surface is rectangular, and viewing ways in which theworkpiece surface is rotated by 180 degrees with respect to an axisviewed directly from the top are the same as each other, the rotationsymmetry number is 2 as an appropriate value. Alternatively, in a casewhere the workpiece surface has a square shape, and viewing ways inwhich the workpiece surface is rotated by 90 degrees with respect to anaxis viewed directly from the top are the same as each other, therotation symmetry number is 4 as an appropriate value. Stillalternatively, in a case where the workpiece surface has a hexagonalshape, and viewing ways in which the workpiece surface is rotated by 30degrees with respect to an axis viewed directly from the top are thesame as each other, the rotation symmetry number is 6 as an appropriatevalue. On the other hand, the circle symmetry is recommended to be setin a case where rotated viewing ways are all the same as each other forany rotation angle, for example, in a case where a workpiece surfacelooks circular when a sphere or a columnar workpiece is viewed directlyfrom the top. If the rotation symmetry is set, in a case whereworkpieces take an attitude for which the rotation symmetry isdesignated, the workpieces are recognized as the same workpiece when asearch model is registered. For example, in a case of a two-foldsymmetry workpiece in which the rotation symmetry number is set to 2, anattitude rotated by 180 degrees is the same as an attitude before beingrotated, and thus the same workpiece is recognized.

If the rotation symmetry is set in the above-described way, a“registration” button 383 is pressed. Consequently, the search model isregistered according to the set conditions. Among the three worksrequired for model registration, the second work and the third work maybe replaced with each other.

Registration of Plural Search Models

If the search model is registered, a search model registration screen390 illustrated in FIG. 94 is displayed. A model list display field 391in which a list of registered search models is displayed is provided inthe operation field 142 of the search model registration screen 390.Herein, a search model having the model number “0” is displayed as aregistered search model along with a reduced image. A state in whichsetting of rotation symmetry is “no” is also displayed for the searchmodel.

If an additional search model is to be registered, an “add” button 392is pressed. Consequently, a search model may be registered according tothe above-described procedures. In a model registration methoddesignation field 393, as a search model registration method, whether amodel obtained by imaging a real workpiece is registered or isregistered from three-dimensional CAD data may be selected. Herein,“registration using a real workpiece” is selected in the modelregistration method designation field 393, and a search model isadditionally registered.

A search model registration screen 410 on which a search model is addedin the above-described way is illustrated in FIG. 95. On the searchmodel registration screen 410 illustrated in FIG. 95, among the sixworkpieces arranged in the image display field 141, the workpiecedisposed on the lower left side is additionally registered as a newsearch model. The additionally registered search model is added as amodel number “1” in the model list display field 391.

Display of Simple Three-Dimensional Search Result

A result of performing a simple three-dimensional search by using aregistered search model may be displayed on the search modelregistration screens in FIGS. 94 and 95. For example, in the exampleillustrated in FIG. 94, the workpiece on the upper left side used forregistration is searched for as a result of performing athree-dimensional search on the six workpieces displayed in the imagedisplay field 141 by using the search model having the model number “0”.A point group matching the search model is displayed on the workpiecesearched for. In this example, in a surface shape of the search model, apoint group on the face is displayed white, and a point grouprepresenting a contour is displayed violet. A score calculated as anevaluation index for the simple three-dimensional search result may bedisplayed in a numerical value. In the example illustrated in FIG. 94,96.4 is displayed as a score value. The reason why a score is not themaximum value (99.9 in this example) for the original workpieceregistered as a search model is that there is a variation between animaging state during model registration and an input image duringexecution of a three-dimensional search, and thus a score is slightlyreduced. It can be recognized that a state matches substantially 96% ormore of a state during model registration from the result of the scorevalue.

Search Region Setting Unit

If the search model is registered in the above-described way, a searchregion is then set in order to execute a three-dimensional searchfunction. Specifically, if a necessary search model is registered, andthen a “completion” button 411 on the lower right side is pressed on thesearch model registration screen 410 in FIG. 95, a search region settingscreen 420 in FIG. 96 is displayed. A search region on which athree-dimensional search is performed is designated on the search regionsetting screen 420. Highlight display transitions from the “modelregistration” icon 352 to the “floor/box setting” icon 353 in the flowdisplay portion 351 on the upper side in the screen, and thus a stage inwhich the model registration is finished, and registration of a floorsurface or a storage container on or in which the workpiece is placed iscurrently performed is displayed to the user.

A search region setting field 421 is provided in the operation field 142of the search region setting screen 420. The search region setting field421 is provided with a “designation method selection navigation” button422 for explaining a method of designating a search region on which athree-dimensional search is performed to the user, and a designationmethod selection field 423 for selecting a method of designating asearch region. If the “designation method selection navigation” button422 is pressed, a navigation screen for showing a method of designatinga search region on which a three-dimensional search is performed to theuser is displayed. On the other hand, in the designation methodselection field 423, one of floor designation, box designation, a boxsearch, and no designation may be selected as an option from a dropbox.Among the options, if the floor designation is selected, a search regionsetting screen 430 in FIG. 97 is displayed, a floor surface designationdialog 431 is displayed, and the explanation that “click three pointsfor calculating a floor surface” is displayed to the user. Consequently,if the user designates any point indicating the floor surface with apointing device such as a mouse on the image display field 141, a spacecoordinate (X,Y,Z) of the designated point is displayed. A gauge isdisplayed as an extraction region. The extraction region indicates arange referred to centering on the clicked position when a Z coordinatevalue at the clicked point is calculated. Data of each point which isthree-dimensionally measured also includes an error, and thus it ispreferable to use a value obtained by using an average value or a medianof data within a predetermined range in order to suppress the influenceof an error. In this case, a Z coordinate value is obtained by using anaverage value in a range designated by the extraction region. If thefloor surface is designated in the above-described way, informationregarding the floor surface is calculated, and the calculatedinformation regarding the floor surface is displayed in a floor surfaceinformation display field 441 of the search region setting field 421 asin a search region setting screen 440 in FIG. 98. Here, in the floorsurface information display field 441, an X inclination, a Yinclination, and a Z intercept of the floor surface are displayed innumerical values as the floor surface information. An offset amount maybe set as necessary. For example, a region corresponding to apredetermined height from the floor surface is removed from the searchregion by taking into consideration a variation or the like in athickness of the floor surface of a storage container. In the exampleillustrated in FIG. 98, a noise removal field 442 is provided to removenoise from the floor surface. Herein, a range from the designated heightof the floor surface to a height of 1.5 mm designated in the noiseremoval field 442 is removed from a three-dimensional search targetregion. If setting of the floor surface is completed in theabove-described way, a “completion” button 443 is pressed. Consequently,the screen is changed to a search parameter setting screen 450 in FIG.99.

Search Parameter Setting Unit

A search parameter is set as a condition for performing athree-dimensional search on the search parameter setting screen 450 inFIG. 99. Highlight display transitions from the “floor/box setting” icon353 to the “search setting” icon 354 in the flow display portion 351 onthe upper side in the screen, and thus a stage in which the searchregion setting is finished, and a search condition is currently set isdisplayed to the user. The operation field 142 of the search parametersetting screen 450 is provided with fields for setting respective searchparameters required to perform a three-dimensional search. Specifically,there are provided a detection condition setting field 451, a featureextraction condition setting field 452, and a determination conditionsetting field 453. Among the fields, in the feature extraction conditionsetting field 452, an edge extraction threshold value which is athreshold value for extracting an edge when a workpiece is detected froman input image is set. In the determination condition setting field 453,an upper limit or a lower limit is set for a score measured value, or anupper limit or a lower limit is set for an X coordinate measured valueof a position, and a determination condition regarding whether or notthe value is to be selected as a search result is set.

In the detection condition setting field 451, a condition for detectinga workpiece is set. In this example, there are provided a detectionnumber designation field 454 for designating an upper limit of thenumber of workpieces to be detected, an inclined angle upper limitdesignation field 455 for designating an upper limit of an inclinedangle of a detected workpiece, and a score lower limit designation field456 for designating a lower limit of a score which is an evaluationindex. The designation may be performed by using a numerical value, andconsecutive adjustment may be performed by using a slider or the like.If a detail setting button 457 is pressed, the screen is changed to asearch parameter setting screen 460 in FIG. 100, and a detection detailcondition setting dialog 461 for setting more detailed detectionconditions is displayed.

The detection detail condition setting dialog 461 is provided with abasic setting field 462, a detection condition detail setting field 463,and an option setting field 464. The basic setting field 462 is providednot only with the detection number designation field 454 but also with asearch sensitivity setting field 465 for setting the sensitivity of athree-dimensional search, and a search accuracy setting field 466 forsetting the accuracy of a three-dimensional search. The detectioncondition detail setting field 463 is provided not only with theinclined angle upper limit designation field 455 and the score lowerlimit designation field 456 but also with a reference angle settingfield 163 a for setting a reference angle and a range setting field 163b for setting an angle range as the rotation angle range setting field163.

A detection condition may be set to be common to search models, and maybe separately for each search model. In the example illustrated in FIG.100, a search model separate designation field 467 is provided, and adetection condition in a three-dimensional search can be separately setfor each search model by checking a checkbox. Such an example isillustrated in a search parameter setting screen 470 in FIG. 101.Herein, a checkbox of the search model separate designation field 467 ischecked, a search model selection field 467 b is displayed, and thus asearch model for which a detection condition is set can be selected.This example shows a state in which the search model A is selected, anddetection conditions regarding the search model A shown in model listdisplay field 391 in FIG. 94 or the like can be separately designated inthe inclined angle upper limit designation field 455, the referenceangle setting field 163 a, the range setting field 163 b, and the scorelower limit designation field 456.

On the other hand, the option setting field 464 illustrated in FIG. 100or the like is provided with a label order designation field 468 fordefining an order of a label for a label number allocated as a serialnumber for identifying a search result, and a determination labeldesignation field 469 for designating a determination label. Adescending order or an ascending order of a correlation value, or adescending order or an ascending order of a Z direction height may beselected in the label order designation field 468. The correlation valuementioned here is the same as the above-described “score”. A target fordetermination of whether or not the target is located at a position oran attitude which is different from an expected position or attitude ina specific search result may be designated in the determination labeldesignation field 469. Herein, the target is designated by using aserial number for identifying a search result. For example, setting isperformed in a case where it is necessary to determine whether or notdata without reliability is obtained as a search result, such as a casewhere a search result shows an unexpected position, rotation angle, orinclined angle.

If three-dimensional search setting is sequentially completed in theabove-described way, an “OK” button 471 provided on the lower right sideof the operation field 142 is pressed, and setting of three-dimensionalsearch conditions is finished.

Registration of Search Model Based on Three-Dimensional CAD Data

In the above-described example, a description has been made of theprocedures of registering a model obtained by imaging a real workpieceas a search model (corresponding to FIG. 62 or the like). However, asdescribed in FIG. 28 or the like, three-dimensional CAD data indicatinga shape of a workpiece, created in advance, may be registered as asearch model. Hereinafter, procedures of registering three-dimensionalCAD data as a workpiece model will be described with reference to FIGS.102 to 106.

First, “register from CAD data (STL format)” is selected with a radiobutton from the model registration method selection field 355 on asearch model registration method selection screen 480 in FIG. 102. Thus,an explanation of procedures of registering three-dimensional CAD dataas a search model is displayed in text and pictures in the operationfield 142. In this example, work to be performed is presented to a userby using text information that “three-dimensional information of aworkpiece surface viewed from positive and negative directions (a totalof six directions) of XYZ axes of three-dimensional CAD data is modeled;only a face desired to be searched for from a generated model isselected; and this tool recognizes a position and an attitude of aworkpiece by matching the registered surface model with the workpiecetaking various attitudes”, and pictures. If registration from CAD datais selected on the search model registration method selection screen 480in FIG. 102, and then an “OK” button 481 is pressed, the screentransitions to a CAD data reading screen in FIG. 103. Specifically, thescreen transitions to a search model registration screen 490, and a CADdata reading dialog 491 is displayed.

CAD Data Reading Unit

In the CAD data reading dialog 491 illustrated in FIG. 103, a storagedestination of read CAD data is designated, and CAD data is furtherselected. As a reading destination, a reading medium such as asemiconductor memory may be selected in addition to the storage unit 9.If a file is selected from a file list display field 492, and an“execution” button 493 is pressed, three-dimensional CAD data is read.

Registration Model Selection Unit

If the three-dimensional CAD data is read, a face registered as a searchmodel is selected. Here, a search model registration screen 510illustrated in FIG. 104 is displayed, and six drawings corresponding tofundamental direction images of a workpiece model are automaticallydisplayed on a registration model selection screen 511. A user mayselect a face desired to be registered in this state. Herein, a checkboxis checked by default, and the checkbox is not checked for a face whichis not required to be registered as a search model, and thus the facecan be excluded from a search model. If a registration target isselected in the above-described way, a “next” button 512 is pressed, andthe screen is changed to a search model registration screen 520 in FIG.105.

Rotation Symmetry Designation Unit

On the search model registration screen 520 in FIG. 105, a rotationsymmetry designation dialog 521 is displayed, and the user may set aface having a symmetric shape even if the face is rotated upward,downward, frontward, rearward, leftward, or rightward with respect to afundamental direction image having rotation symmetry among fundamentaldirection images selected as registration targets. If setting ofrotation symmetry is completed in the above-described way, a“registration” button is pressed, and a search model is registered.

If the search model is registered, registered search models aredisplayed in a list form in the model list display field 391 as in thesearch model registration screen 530 in FIG. 106. A plurality of searchmodels may be registered for a single workpiece as described above. Forexample, in the example illustrated in FIG. 106, six search models A, B,C, D, E and F illustrated in FIG. 105 or the like are registered for athree-dimensional CAD workpiece model having the model number “0”.Therefore, the search models A to F for the model number “0” aredisplayed in a list form in the model list display field 391. Searchmodels which cannot be displayed on one screen in the model list displayfield 391 may be displayed to be scrolled for switching display.

Search Model Editing Unit

A registered search model may be edited. In the example illustrated inFIG. 106, if an “edit” button 531 of the operation field 142 is pressed,a model editing screen 620 illustrated in FIG. 114 is displayed. On themodel editing screen 620, a registered search model is displayed in theimage display field 141, and set items are displayed in the operationfield 142. Herein, a model region setting field 621, a backgroundremoval setting field 327 and a rotation symmetry setting field 382 as afeature setting field 622, and a model number display field as a modelregistration field 623 are provided. The user may select an item desiredto be changed, and change the item as appropriate.

Model Region Detail Setting Unit

If a detail setting button 624 provided in the model region settingfield 621 is pressed, a model region detail setting screen 630illustrated in FIG. 115 is displayed. On the model region detail settingscreen 630, a mask region can be set as a region desired to be excludedfrom a search model registration target in a range of the model region362 displayed in the image display field 141. The operation field 142 isprovided with the model region designation field 621, a mask regiondesignation field 631, and a weight region designation field 632. Aplurality of mask regions may be set in the mask region designationfield 631. The mask region may be set to a predefined shape such as arectangular shape or a circular shape, and may also be set throughautomatic detection of a free curve or a boundary portion. A weight maybe set in the weight region designation field 632 separately from eachmask region.

When a search result is scored, if there is a location of a featurepoint desired to be focused more than other feature points included in asearch model, a score result may be adjusted by surrounding the locationwith a weight region. For example, in a case where there is a shapecausing a search model to be specified or causing an attitude in arotation direction to be defined depending on the presence or absence ofa feature of a part of the entire shape, a weight region is set for theimportant feature, and thus it is possible to improve detectionperformance in a search. A weight extent for other feature points may bedesignated by a ratio. For example, in a case where a weight extent isset to “3”, adjustment may be performed such that a score value iscalculated with the influence of three times the influence of otherfeature points.

Feature Detail Setting Unit

If a detail setting button 625 provided in the feature setting field 622is pressed, a feature detail setting screen 640 illustrated in FIG. 116is displayed. On the feature detail setting screen 640, a feature or thepresence or absence of rotation symmetry is set for the model region 362displayed in the image display field 141. The operation field 142 isprovided with a feature detail setting field 641 and a rotation targetsetting field 382. The feature detail setting field 641 is provided witha feature extraction interval designation field 642 for designating afeature extraction interval, a background removal setting field 327, andan edge extraction threshold value setting field 643. A featureextraction interval is a parameter influencing an extraction interval ofa feature point when a search model is generated, and any one of“accuracy priority”, “standard”, and “speed priority” may be selected.In a case where the “accuracy priority” is selected, an extractioninterval of feature points in a search model is narrow, a finer featurecan be recognized, and thus detection accuracy is increased, but aprocess time is increased due to an increase in a data amount. On theother hand, in a case where the “speed priority” is selected, anextraction interval of feature points in a search model is wide, a finerfeature is omitted, and thus detection accuracy is lowered, but aprocess time is reduced due to a decrease in a data amount. The featureextraction interval may be adjusted depending on a shape or a size of aworkpiece to be used, or allowable process time or detection accuracy.

In the above-described way, the user can adjust a necessary item suchthat a more appropriate search result is obtained while referring to asimple three-dimensional search result, an obtained score, or the like.If the search model is registered in the above-described way, followingprocedures subsequent thereto are the same as the procedures (searchregion setting and search parameter setting) of registering a searchmodel by imaging a real workpiece described in FIGS. 96 to 101, and thusdescription thereof will be omitted.

The search model is registered in advance in the above-described way,and then a three-dimensional search is performed. The three-dimensionalsearch is used for an actual operation, that is, used to perform binpicking of imaging a bulk workpiece group in the sensor unit andspecifying a grip position of a workpiece which can be gripped, and isalso used in a case where simulation of three-dimensional picking isperformed in advance. The simulation of three-dimensional picking is anoperation of determining whether or not a workpiece can be gripped by anend effector at a grip position designated for a workpiece model inadvance by the grip position specifying portion 8 d on the basis of eachsearch result obtained through a search in the three-dimensional searchportion 8 k.

Search Result Display Unit

Next, a description will be made of an example of performing athree-dimensional search on workpieces loaded in bulk with reference toFIGS. 107 to 113. FIG. 107 illustrates an example of a search resultdisplay screen 540 showing a result of performing a three-dimensionalsearch on a bulk workpiece group. The search result display screen 540is provided with an image display field 141 and an operation field 142,and a list of search models used for the three-dimensional search isdisplayed in a vertical direction in the model list display field 391 ofthe operation field 142. On the other hand, in the image display field141, workpieces obtained as a search result are displayed to overlap theworkpieces in bulk included in an input image, and a point group whichis a search result and a score which is an evaluation index therefor aredisplayed.

Search Result List Unit

A search result list display field 541 is provided on an upper part ofthe image display field 141, and a list of search models used and scoresis displayed for search results obtained through the search. Herein, thenumber of workpieces searched for, the presence or absence of aworkpiece, a total volume/a volume of a workpiece, a model number or ascore of a search result having a specific label number, and a position(X,Y,Z) are displayed. The total volume/a volume of a workpieceindicates a value obtained by dividing a total volume of a point grouplocated at a higher position than a floor surface or a bottom of areturnable box by an average volume expected from a registered searchmodel, and is a data indicating the number of workpieces present in aninput image. In a case where a numerical value thereof is greater than apredetermined numerical value, a result of the presence or absence of aworkpiece is determined as being presence (1). By checking whether aresult of the presence or absence of a workpiece is presence or absencein a case where the number of workpieces searched for is 0, and thus agrip solution is not obtained, it is possible to check whether pickingis finished in a state there is no workpiece or in a state in which aworkpiece remains.

Display items in the search result list display field 541 may switch.For example, a switching button 542 for switching the display content isprovided on the lower right side in the search result list display field541 of the search result display screen 540 illustrated in FIG. 107. Theswitching button 542 is formed of left and right arrows, and the numberof screens 543 is displayed on the upper part thereof. In this example,as the number of screens 543, “1/11” indicating the first screen ofeleven screens is displayed, an upper limit is imposed on a number bypressing the switching button 542, and thus the display content of thesearch result list display field 541 can switch. For example, if thenumber of screens 543 is changed to “2/11” by operating the switchingbutton 542, the screen is changed to a search result display screen 550illustrated in FIG. 108, and a summary of a three-dimensional searchresult is displayed in the search result list display field 541. Herein,an inclined angle, a rotation angle, an attitude (R_(X),R_(Y),R_(Z)),and the like are displayed in addition to the number of workpiecessearched for, the presence or absence of a workpiece, and the totalvolume/a volume of a workpiece. If the number of screens 543 is changedto “3/11”, the screen is changed to a search result display screen 560illustrated in FIG. 109, and inclinations of XYZ or the like aredisplayed as information regarding a floor surface in addition to thenumber of workpieces searched for, the presence or absence of aworkpiece, and the total volume/a volume of a workpiece as the summaryof a three-dimensional search result in the search result list displayfield 541.

If the number of screens 543 is changed to “4/11”, the screen is changedto a search result display screen 570 illustrated in FIG. 110, and modelnumbers of search models used for detection of search results and scoresare horizontally displayed for each label number for identifying asearch result as a summary of a three-dimensional search result in thesearch result list display field 541. If the number of screens 543 ischanged to “5/11”, the screen is changed to a search result displayscreen 580 illustrated in FIG. 111, and a position (X,Y,Z) is displayedfor each label number for identifying a search result as a summary of athree-dimensional search result in the search result list display field541. If the number of screens 543 is changed to “6/11”, the screen ischanged to a search result display screen 590 illustrated in FIG. 112,and an inclined angle and a rotation angle are displayed for each labelnumber for identifying a search result as a summary of athree-dimensional search result in the search result list display field541. If the number of screens 543 is changed to “7/11”, the screen ischanged to a search result display screen 610 illustrated in FIG. 113,and an attitude (R_(X),R_(Y),R_(Z)) is displayed for each label numberfor identifying a search result as a summary of a three-dimensionalsearch result in the search result list display field 541.

Three-Dimensional Picking

Next, with reference to FIGS. 117 to 149, a description will be made ofprocedures of performing simulation of three-dimensional picking fordetermining whether or not a workpiece can be gripped by an end effectoron the basis of each search result obtained through a three-dimensionalsearch. If the “3D pick” button 343 is pressed on the function selectionscreen 340 in FIG. 89, a three-dimensional picking initial settingscreen 650 in FIG. 117 is displayed, and a guidance function is executedsuch that a user sequentially performs setting required to performthree-dimensional picking. Herein, a 3D picking guidance field 655 isdisplayed on the upper side in the operation field 142, and worksequentially performed is displayed as an icon. In this example, fourprocesses such as “initial setting”, “grip registration”,“condition/verification”, and “place setting” are displayed.

Three-Dimensional Picking Initial Setting Screen

First, initial setting for performing three-dimensional picking isperformed on the three-dimensional picking initial setting screen 650 inFIG. 117. An “initial setting” icon is highlighted in the 3D pickingguidance field 655 on the upper side in the operation field 142, andwork to be performed in this stage is displayed to the user. Theoperation field 142 is provided with a calibration data selection field651, a detection tool setting field 652, and a hand registration field653. In the calibration data selection field 651, calibration datacalculated in advance for converting a coordinate system displayed bythe robot setting apparatus into a coordinate system in which the robotcontroller operates an end effector is called and registered. In thedetection tool setting field 652, means for detecting a workpiece isdesignated. In this example, a workpiece detection three-dimensionalsearch set to a tool number “100” is selected. The above-describedsetting of the search model registered through the setting of thethree-dimensional search is read. In a case where a search model of adiffering workpiece has been set to a separate tool number, selection ofthe detection tool may be changed such that a setting target ofthree-dimensional picking is changed.

End Effector Model Setting Unit

An end effector model is registered in the hand registration field 653.Here, if a “hand model creation” button 654 is pressed, an end effectormodel setting screen 660 illustrated in FIG. 118 is displayed. On theend effector model setting screen 660, a list of registered end effectormodels is displayed in a hand model list display field 661. In theexample illustrated in FIG. 118, an end effector model is notregistered, and thus an end effector model is not displayed in the imagedisplay field 141 and the end effector model list display field 661 ofthe operation field 142. Therefore, an “add” button 662 is pressed, andan end effector model is added. The flange surface FLS is displayed as areference plane of the arm portion tip of the robot at which an endeffector model is disposed in the image display field 141 of the endeffector model setting screen 660. Tool coordinate axes TAX indicatingcoordinate axes of the end effector model EEM side are also displayed(details thereof will be described later).

End Effector Model Editing Unit

In FIG. 118, if an “add” button 662 is pressed, an end effector modelediting screen 670 illustrated in FIG. 119 is displayed. Parts formingthe end effector model are set on the end effector model editing screen670. Herein, registered parts are displayed in a parts list displayfield 671. In the example illustrated in FIG. 119, parts are notregistered yet, and thus nothing is displayed in the parts list displayfield 671. If an “add” button 672 is pressed in this state, a partsadding dialog illustrated in FIG. 120 is displayed, and whether parts tobe added are registered by reading CAD data or are newly created byusing a cuboid model or a cylindrical model is selected with a radiobutton. Additional parts are an aspect of the above-described additionalregion, are not limited to a cuboid or a cylinder, and may use any shapesuch as a prismatic shape, a triangular shape, or a spherical shape. Inthe example illustrated in FIG. 120, CAD data is selected. If createdthree-dimensional CAD data is selected by pressing an “add” button 681,a CAD position/attitude setting screen 690 illustrated in FIG. 121 isdisplayed, three-dimensional CAD data is read such that a CAD model CDMcreated with CAD is displayed in the image display field 141, and aposition/attitude field 691 for designating a position and an attitudeof the CAD model CDM is displayed in the operation field 142. A positionand an attitude of the CAD model CDM are designated with the flangesurface FLS as a reference. The user designates an attachment positionor angle of the end effector model from the position/attitude field 691.For example, if 200 is designated as the rotation angle R_(X) in thestate illustrated in FIG. 121, the CAD model CDM displayed in the imagedisplay field 141 is displayed at an attitude rotated by 20° centeringon the X axis as illustrated in FIG. 122. Alternatively, if 50 mm isdesignated in the Z direction in the state illustrated in FIG. 121, theCAD model CDM is displayed to be offset by 50 mm in the Z direction asillustrated in FIG. 123. If the position and an attitude of the CADmodel CDM are adjusted in the above-described way, and then an “OK”button 692 in FIG. 121 is pressed, the CAD data is registered, and isdisplayed in the parts list display field 671.

A plurality of parts may be added. For example, if an “add” button 682is pressed on the end effector model editing screen in FIG. 124, acylindrical model is selected in the part adding dialog 680 in FIG. 120,and an “add” button 681 is pressed, an additional regionposition/attitude setting screen illustrated in FIG. 125 is displayed,and a cylindrical model is displayed as a new additional region ADA inthe image display field 141. A part which is being edited ishighlighted. In the example illustrated in FIG. 125, the additionalregion ADA is displayed red, and the CAD model CDM is displayed gray orwith a wire frame. Consequently, the user can easily visually recognizea part which is currently edited. Similarly, a position and an attitudeof the additional region ADA to be added are designated from theposition/attitude field 691. In this example, a Z position is designatedto be −50 mm in the position/attitude field 691, and thus the additionalregion ADA is disposed in an opposite direction to the CAD model CDMfrom the flange surface FLS. A radius or a height of the cylindricalmodel as a size of the additional region ADA may be designated in a sizedesignation field 693. If the conditions for the additional region ADAare set in the above-described way, and then an “OK” button 692 ispressed, the additional region ADA is registered, and the cylinder isadded and displayed in the parts list display field 671 of the endeffector model editing screen as illustrated in FIG. 126.

Tool Coordinate Designation Unit

A position and an attitude of a registered part may be adjusted. Forexample, if a part to be selected is changed from the additional regionADA to the CAD model CDM in the image display field 141 or the partslist display field 671 on the end effector model editing screen in FIG.126, a screen illustrated in FIG. 127 is displayed, and the user mayadjust a position (X,Y,Z) and an attitude (R_(X),R_(Y),R_(Z)) in a toolcoordinate designation field 673 in this state. For example, asillustrated in FIG. 128, if a Z position is set to 145 mm, the toolcoordinate axes TAX are moved in the Z direction from the flange surfaceFLS by the designated distance, and are displayed in the image displayfield 141. If the rotation angle R_(X) is set to 200 in this state asillustrated in FIG. 129, the tool coordinate axes TAX are rotatedclockwise by 20° centering on the X axis, and are display.

A plurality of parts are registered in the above-described way, and theend effector model EEM is built by combining the parts with each other.In a case where setting of each part is completed, if an “OK” button 674is pressed on the end effector model editing screen in FIG. 128 or thelike, the plurality of parts are combined to be registered as the endeffector model EEM, and the end effector model EEM is displayed as “hand0” in the hand model list display field 661 as illustrated in the endeffector model setting screen 660 in FIG. 130. The parts displayed inpieces are displayed as the integrated end effector model EEM in theimage display field 141.

In this example, a description has been made of an example of formingthe end effector model by combining parts with each other, but a part isnot limited to an end effector model or a part thereof, and, forexample, a cable or the like may be expressed as a part. As mentionedabove, a part is an aspect of an additional model expressing a part ofan end effector model, an object added to a surface thereof, or aperipheral member.

Grip Reference Point HBP

The grip reference point HBP of the end effector model EEM is furtherdisplayed in the image display field 141. The grip reference point HBPindicates a position at which the end effector model EEM grips aworkpiece. For example, the grip reference point HBP is set to thecenter between claws of the end effector model EEM gripping a workpiece.The grip reference point HBP may be calculated on the robot settingapparatus side so as to be determined according to a predetermined rule,and may be designated to any position by a user.

The tool coordinate axes defining a position and an attitude of the endeffector model EEM are preferably further displayed in the image displayfield 141 in an overlapping manner. The tool coordinate axes arepreferably rotation-completed coordinate axes RAX which are changedaccording to rotation of the end effector model EEM. The origin of therotation-completed coordinate axes RAX more preferably matches the gripreference point HBP. Consequently, a user can easily visually recognizea state of the end effector model EEM when a grip position is set.

Grip Registration Unit

If the end effector model EEM is registered as initial setting in theabove-described way, a grip position is then registered. On a gripregistration screen 710 in FIG. 131, highlight display in the 3D pickingguidance field 655 transitions from the “initial setting” icon to the“grip registration” icon. The operation field 142 is provided with asearch model switching field 711 for selecting a registered searchmodel, and a grip registration list display field 712 displaying a listof grip positions registered for a search model selected in the searchmodel switching field 711. As described above, in this example, aworkpiece model for which a grip position is registered is the same as asearch model used for a three-dimensional search. Herein, it is assumedthat the six search models A to F for the model number “0” registered inFIG. 104 are registered as registered search models in the search modelswitching field 711. In the example illustrated in FIG. 131, since agrip position is not registered yet, nothing is displayed in the gripregistration list display field 712. The search model A is selected inthe search model switching field 711, and thus a search model SMAcorresponding to the search model A is displayed in the image displayfield 141. The reference coordinate axes BAX which are coordinate axesindicating a position and an attitude of the search model SMA aredisplayed in an overlapping manner. In this state, a user registers agrip position for the search model A. Specifically, if an “add” button713 is pressed, the screen is changed to a grip registration screen in132, and a grip setting dialog 720 is displayed.

Grip Setting Unit

The grip setting dialog 720 of the operation field 142 is provided witha basic setting field 721, a grip position coordinate designation field724, and a movement amount setting field 727. The basic setting field721 includes a grip label setting field 722 and a hand selection field723. In the grip label setting field 722, a grip label which isidentification information for specifying a grip position is set.Herein, “A” indicating a search model and a serial number “000” of agrip position are set as a grip label. In the hand selection field 723,the end effector model EEM gripping a workpiece at the grip position setin the grip label setting field 722 is selected. In this example, the“hand 0” is selected in the hand selection field 723, and thus the endeffector model EEM is displayed in the image display field 141. Asmentioned above, the grip position is set in correlation with the endeffector model EEM.

The grip position coordinate designation field 724 is provided with acoordinate attitude designation field 725 in which a user can directlydesignate a coordinate position (X,Y,Z) and an attitude(R_(X),R_(Y),R_(Z)) indicating a grip position in numerical values, anda “simple setting navigation” button 726 for executing a guidancefunction for showing setting of a grip position to the user.

In the movement amount setting field 727, a direction in which the endeffector model EEM is moved is defined. Herein, an approach movementamount in which the end effector model EEM comes close to the searchmodel SMA in order for the end effector model EEM to grip the searchmodel SMA is defined as distances in the X direction, the Y direction,and the Z direction. In the example illustrated in FIG. 132, an examplein which the end effector model EEM is moved by 100 mm in the Zdirection is illustrated.

Grip Position Setting Guidance Function

If the “simple setting navigation” button 726 is pressed on the screenin FIG. 132, a grip position setting guidance function is executed.Specifically, as illustrated in FIG. 133, a setting method selectiondialog 730 is displayed, and a user selects whether a grip position isset by using a three-dimensional viewer or a robot with a radio button.In this example, the user selects setting on the three-dimensionalviewer. If an “OK” button 731 is pressed in this state, a registrationguidance function on the three-dimensional viewer is executed.

Registration Guidance Function on Three-Dimensional Viewer

In the registration guidance function on the three-dimensional viewer, agrip position is registered through four processes. Herein, the userdesignates a grip position on three-dimensional viewer registrationscreens illustrated in FIGS. 134 to 141. Specifically, if the “OK”button 731 is pressed in FIG. 133, the screen is changed to an X-Ydesignation screen 740 in FIG. 134.

First Process: X-Y Designation

In FIG. 134, the X-Y designation screen 740 is displayed as an aspect ofthe X-Y designation unit. Herein, in the first process for registering agrip position, the search model SMA is displayed in a plan view, and, inthis state, a grip position is designated on the plan view.Specifically, the search model SMA is two-dimensionally displayed in anXY plane in the image display field 141, and X and Y coordinatesregarding a grip position are designated on the search model SMA. Workto be performed is explained to the user in text or the like as STEP1 inthe operation field 142. Herein, the content that “select XY positionsregarding a grip position on a camera image” is displayed. In theexample illustrated in FIG. 134, a height image (search model SMA) ofthe search model A is enlarged and displayed in the image display field141, and, in this state, the user selects a grip designation position P1by using a pointing device such as a mouse. XY coordinates are input toan XY-coordinate designation field 741 according to the grip designationposition P1. An X coordinate and a Y coordinate in the XY-coordinatedesignation field 741 are adjusted, and thus the grip designationposition P1 in the image display field 141 is also updated. On the X-Ydesignation screen 740, only an X coordinate and a Y coordinate ofposition parameters can be designated, in other words, other positionparameters cannot be designated. As mentioned above, guidance is givensuch that a situation in which the user is confused due to a pluralityof position parameters being able to be simultaneously set as in therelated art is prevented, and the user accurately understands a positionparameter which can be set by restricting the position parameter whichcan be set, and sequentially sets position parameters. If designation ofthe XY coordinates is completed in the above-described way, a “next”button 742 is pressed. Consequently, a second process is performed, andthe screen is changed to a Z-R_(Z) designation screen 750 in FIG. 135.

Second Process: Z-R_(Z) Designation

The Z-R_(Z) designation screen 750 in FIG. 135 is an aspect of theZ-R_(Z) designation unit, and a screen allowing a user to designate a Zcoordinate and an attitude angle R_(Z). Work to be performed by the userin a second process is displayed in text in the operation field 142 andis explained. Herein, the content that “STEP2 designate a Z positionregarding a grip position, and an attitude angle R_(Z) of the hand” isdisplayed. A Z designation field 752 for designating a Z position and anR_(Z) designation field 753 for designating an R_(Z) attitude areprovided as a Z-R_(Z) designation field 751 in the operation field 142.The end effector model EEM and the search model SMA arethree-dimensionally displayed in the image display field 141. In otherwords, the image display field 141 functions as a three-dimensionalviewer which three-dimensionally displays an end effector model or asearch model. A numerical value in the Z designation field 752 or theR_(Z) designation field 753 is changed, and thus the three-dimensionaldisplay content of the end effector model EEM or the search model SMA inthe image display field 141 is updated. A viewpoint of thethree-dimensional viewer can be changed by the user dragging the imagedisplay field 141, and thus a grip setting state can be checked indetail.

Only the Z axis is displayed as a coordinate axis regarding a positionparameter which is currently set among the coordinate axes in the imagedisplay field 141 which is a three-dimensional viewer. The Z axis is thecorrection rotational Z axis AXZ after being corrected, displayed withthe above-described Z-Y-X system Euler's angle. As mentioned above,since only a coordinate axis regarding a position parameter which can becurrently adjusted is displayed, and other position parameters cannot beadjusted on this screen, appropriate guidance given to the user isrealized such that only an appropriate position parameter issequentially set according to a predetermined order without causingwasteful confusion in the user.

In the example illustrated in FIG. 135 or the like, an axis (thecorrection rotational Z axis AXZ) in a grip direction is displayed toextend through the grip reference point HBP, and thus the user caneasily understand a movement direction for making the end effector modelEEM come close to a workpiece model. A movement direction of the endeffector model may be indicated by an arrow. In this case, a movementdirection of the end effector model may be displayed to overlap the axisin the grip direction, and may be displayed at a separate position.

The “fit” button 154 is provided as an aspect of the relative positionsetting unit on the Z-R_(Z) designation screen 750. As described above,a fitting function of automatically moving an end effector model to agrip position of a workpiece model is realized by pressing the “fit”button 154. If the “fit” button 154 is pressed, as illustrated in FIG.136, the end effector model EEM is moved to be brought into contact withthe search model SMA. In other words, the grip reference point HBP ofthe end effector model EEM is moved toward the grip designation positionP1 of the search model SMA. An initial position of the end effectormodel EEM when displayed is preferably adjusted in the state illustratedin FIG. 135 such that the movement direction matches the Z axis.Consequently, since the end effector model EEM is moved only in the Zdirection so as to be brought into contact with the search model SMA,intuitively easily understandable display can be performed, and thus auser can easily perform work of moving the end effector model EEM.

A Z position in the Z designation field 752 of the operation field 142is also changed due to movement of the end effector model EEM in theimage display field 141. In this example, in FIG. 135, 200 mm is changedto 50 mm. Herein, when the end effector model EEM is moved toward thegrip designation position P1 designated in FIG. 134 in the heightdirection, that is, the Z axis direction due to the fitting function,the end effector model EEM is moved to a position returned by apredetermined offset amount from a position brought into contact withthe search model SMA with the contact position as a reference. Among thecoordinate axes with the grip designation position P1 as a reference,only the Z axis which is currently set is displayed in the image displayfield 141.

In this state, the user designates the attitude angle R_(Z) in the R_(Z)designation field 753. At the attitude of the end effector model EEMillustrated in FIG. 136, a side surface of the search model SMA inclinedin a C shape is gripped, and thus adjustment is performed such that aparallel surface is gripped by rotating the end effector model EEM inthe Z axis direction. In this case, the user may directly input arotation angle to the R_(Z) designation field 753, and may more easilydesignate rotation of the end effector model EEM by providing an angleselection button 754 for inputting a predetermined angle. In thisexample, four angle buttons of 0°, 90°, 180°, and −90° are disposed asthe angle selection button 754. If the user presses the 90° button, 90°is input to the R_(Z) designation field 753, and thus the end effectormodel EEM is rotated by 90°in the Z axis direction as illustrated inFIG. 137 and is corrected to be able to accurately grip the search modelSMA.

Third Process: R_(Y) Designation

If the Z-R_(Z) designation is completed in the second process in theabove-described way, a “next” button 755 is pressed, and thus R_(Y)designation in the third process is performed. Specifically, if the“next” button 755 is pressed, an R_(Y) designation screen 760 in FIG.138 is displayed. The R_(Y) designation screen 760 is an aspect of theR_(Y) designation unit, and is used to designate an attitude angleR_(Y). An R_(Y) designation field 761 is displayed in the operationfield 142 of the R_(Y) designation screen 760, and the content that“STEP3 designate the attitude angle R_(Y) of the hand” is displayed tothe user as work to be performed in the third process. Among thecoordinate axes, only the correction rotational Y axis AXY which iscurrently set is set in the image display field 141. In this state, theuser designates the attitude angle R_(Y) on the R_(Y) designation screen760 according to an attitude of the end effector model EEM gripping thesearch model SMA. Alternatively, the end effector model EEM is rotatedthrough dragging on the image display field 141 which is athree-dimensional viewer. For example, if 90°is designated as theattitude angle R_(Y) on the R_(Y) designation screen 760 in the stateillustrated in FIG. 138, the screen is changed to an R_(Y) designationscreen in FIG. 139, and the end effector model EEM is displayed at anattitude rotated by 90° centering on the Y axis. If the R_(Y)designation is completed in the third process in the above-describedway, a “next” button 762 is pressed, and thus R_(X) designation in thefourth process is performed.

Fourth Process: R_(X) Designation

FIG. 140 illustrates an R_(X) designation screen 770 in the fourthprocess. The R_(X) designation screen 770 is an aspect of the R_(X)designation unit, and is provided with an R_(X) designation field 771for designating an attitude angle R_(X). In the image display field 141,the end effector model EEM and the search model SMA are displayed tooverlap each other, and, among the coordinate axes, the correctionrotational X axis AXX which is the X axis regarding a position parameterR_(X) which can be set on the R_(X) designation screen 770 is displayed.In this state, the user designates the attitude angle R_(X) in the R_(X)designation field 771. Alternatively, the end effector model EEM isrotated through dragging on the image display field 141. For example, inthe example illustrated in FIG. 140, 180° is input to the R_(X)designation field 771 as the attitude angle R_(X), so as to correspondto the current grip position, and, if 150° is designated, attitudes ofthe end effector model EEM and the search model SMA are changed in theimage display field 141 which is a three-dimensional viewer according tothe designated attitude angle R_(X) as illustrated in FIG. 141.

If the attitude angle R_(X) is designated in the above-described way, a“completion” button 772 is pressed, and registration of the gripposition is finished. If the grip position is registered, as illustratedin FIG. 142, grip registration A-000 is added to the grip registrationlist display field 712. The search model A corresponding to the gripregistration A-000 is highlighted in the search model switching field711. A scene in which the search model SMA is gripped at the positionand the attitude of the end effector model EEM for which the gripregistration has been performed is three-dimensionally displayed in theimage display field 141. The reference coordinate axes BAX of the searchmodel SMA side are displayed as coordinate axes in an overlappingmanner.

In the above-described way, other grip positions are registered for thesearch model A, or grip positions are added to the other search models Bto F. A grip position may be added by pressing the “add” button 713provided in the grip registration list display field 712 as describedabove. In an example illustrated in FIG. 143, an example in which gripregistration B-000 is registered for the search model B is illustrated.

Copy Function

A copy function of copying a registered grip position may be used toadditionally register a grip position. If a “copy” button 714 providedin the grip registration list display field 712 on the screen in FIG.142 or the like is pressed, a registered grip position can be copied.Correction is performed on the basis of the copied grip position, andthus registration work for a grip position can be efficiently performed.If an “edit” button 715 is pressed in the state illustrated in FIG. 142or the like, a grip setting dialog is displayed, and thus a gripposition can be set as described above. Here, as described above, the“simple setting navigation” button 726 may be pressed such that the gripposition setting guidance function is executed, and a necessary locationmay be corrected.

Condition Verification Screen 780

If grip positions are registered for the search models A to F in theabove-described way, a “completion” button 716 is pressed on the screenin FIG. 143 or the like such that the grip registration process isfinished, and a condition verification process is performed. In thecondition verification process, a three-dimensional search is performedon a bulk workpiece group such that a workpiece is detected, and it isverified whether or not the workpiece can be gripped by an end effectorat a designated grip position. Herein, simulation is performed by usingthe end effector model EEM and a search model. FIG. 144 illustrates anexample of a condition verification screen 780. On the conditionverification screen 780, highlight display in the 3D picking guidancefield 655 transitions from the “grip registration” icon to the“condition/verification” icon. The operation field 142 is provided witha detection condition setting field 781 for setting a condition fordetecting a grip position on the basis of a three-dimensional searchresult, and a verification field 785 for verifying grip propriety of adetected workpiece. The detection condition setting field 781 includes adetection number designation field 782 for designating an upper limit ofthe number of grip solutions to be detected, a hand inclined angle upperlimit designation field 783 for designating an upper limit of aninclined angle of an end effector model, and a margin setting field 784for setting a margin from a wall surface of a storage container in orderto expand an interference range of an end effector model. If the endeffector model EEM enters a range of a distance (5 mm in FIG. 144)designated in the margin setting field 784, interference is determined.

On the other hand, the verification field 785 is provided with adetection number display field 786 indicating the number of searchresults which are actually detected as a result of simulation of athree-dimensional search, a display label designation field 787 fordesignating a label number of a search model desired to be verifiedamong detected search results, and a “verification for each workpiece”button 788 for performing verification on each workpiece. A searchresult having a label number designated in the display label designationfield 787 is displayed in the image display field 141.

Verification on Each Workpiece

If the “verification for each workpiece” button 788 is pressed, averification dialog 790 illustrated in FIG. 145 is displayed. Theverification dialog 790 is an aspect of the workpiece selection screen210 described in FIG. 74. In the verification dialog 790 in FIG. 145, itis verified whether or not each detected workpiece can be gripped. Inthis example, the verification dialog 790 includes a target workpieceselection field 791, a detection search model display field 792, and a“grip check” button 793. If the “grip check” button 793 is pressed, adetection result display dialog 810 illustrated in FIG. 146 isdisplayed. The detection result display dialog 810 is an aspect of thegrip solution candidate display screen 220 described in FIG. 75 or thelike. Herein, determination results of grip propriety at a set gripposition for each face (a search image and an estimation image) detectedfor a workpiece (the second workpiece in FIG. 145) selected in thetarget workpiece selection field 791 are displayed in a list form in agrip solution candidate display field 811. In the example illustrated inFIG. 146, since a grip label C-000 is selected, the end effector modelEEM is disposed and displayed at a corresponding grip position in theimage display field 141, and the end effector model EEM is colored anddisplayed according to a determination result. In this example, adetermination result is poor grip, and thus the end effector model EEMis displayed red. A point group interference error is displayed as acause of grip impossibility in the grip solution candidate display field811. Consequently, the user is provided with a guideline for necessarymeasures, for example, correction or addition of a grip position.

If the grip label A-000 for which a determination result is good grip isselected in a detection result display dialog in FIG. 147, the endeffector model EEM gripping a workpiece at a corresponding grip positionis thus displayed white in the image display field 141. In theabove-described way, it is possible to determine grip propriety, andalso to examine an appropriate measure as necessary.

If a detection condition detail setting button 789 provided in thedetection condition setting field 781 is pressed on the conditionverification screen 780 in FIG. 144, the screen is changed to adetection condition detail setting screen 820 in FIG. 148. The detectioncondition detail setting screen 820 is provided with a grip positiondetection condition setting field 821, an interference determinationsetting field 822 for determining interference between an end effectorand a box or a floor, and an interference determination setting field823 for determining interference between an end effector and athree-dimensional point group.

If the condition verification process is completed in theabove-described way, a place setting process of defining a placementposition of a workpiece is finally performed. FIG. 149 illustrates anexample of a place setting screen 830. In the 3D picking guidance field655 of the place setting screen 830, highlight display transitions fromthe “condition/verification” icon to the “place setting” icon. Theoperation field 142 includes a tool coordinate setting field 831 and aplace position registration field 832. The user performs settingregarding a place of a workpiece on this screen.

The robot setting apparatus, the robot setting method, the robot settingprogram, the computer readable recording medium, and the apparatusstoring the program according to the present invention can beappropriately used to verify a bin picking operation of a robot.

What is claimed is:
 1. A robot setting apparatus setting an operation ofa robot performing a bin picking operation of a sensor unit measuring athree-dimensional shape of each of a plurality of workpieces stacked ina work space and sequentially taking out the workpieces with an endeffector provided at a tip of an arm portion of the robot, the robotsetting apparatus comprising: a workpiece model registration unit thatregisters a workpiece model virtually expressing a three-dimensionalshape of a workpiece with three-dimensional CAD data or a height image;an end effector model registration unit that registers an end effectormodel virtually expressing a three-dimensional shape of an end effectorwith three-dimensional CAD data; an image display region in which theend effector model and the workpiece model are displayed on a virtualthree-dimensional space; a grip reference point setting unit thatdefines a grip reference point corresponding to a position at which theworkpiece model is gripped for the end effector model; a grip directionsetting unit that defines a grip direction in which the end effectormodel grips the workpiece model; a workpiece side grip locationdesignation unit that designates a grip position at which the endeffector model grips the workpiece model in a state in which at leastthe workpiece model is displayed in the image display region; and arelative position setting unit that sets a relative position between theend effector model and the workpiece model such that the grip directiondefined in the grip direction setting unit is orthogonal to a workpieceplane representing an attitude of the workpiece model displayed in theimage display region, and the grip reference point is located at thegrip position along the grip direction.
 2. The robot setting apparatusaccording to claim 1, wherein the relative position setting unitautomatically adjusts the relative position between the end effectormodel and the workpiece model such that the grip direction is orthogonalto the workpiece plane, and the grip reference point and the gripposition are located on an axis along the grip direction.
 3. The robotsetting apparatus according to claim 1, wherein the relative positionsetting unit moves the end effector model along the grip direction untilthe end effector model interferes with the workpiece model, andautomatically defines a grip state at an attitude of being separatedfrom a position reaching an interference position by a predetermineddistance, in a state in which the grip direction is orthogonal to theworkpiece plane, and the grip reference point and the grip position arelocated on an axis along the grip direction by adjusting the relativeposition between the end effector model and the workpiece model.
 4. Therobot setting apparatus according to claim 1, further comprising: asearch model registration unit that registers a search model which isused to perform a three-dimensional search for specifying an attitudeand a position of each workpiece included in an input image from theinput image indicating a state in which a plurality of workpiece groupsare loaded in bulk, and which virtually expresses a three-dimensionalshape of a workpiece; a three-dimensional search unit that performs athree-dimensional search for specifying an attitude and a position ofeach workpiece from the input image by using the search model registeredby the search model registration unit; and a three-dimensional pickdetermination unit that determines whether or not a workpiece can begripped by an end effector at a grip position designated for theworkpiece by the workpiece side grip location designation unit on thebasis of a search result in the input image searched by thethree-dimensional search unit.
 5. The robot setting apparatus accordingto claim 4, further comprising: an input image acquisition unit thatacquires an input image including a three-dimensional shape on the basisof an image of a plurality of workpiece groups measured in the sensorunit, wherein the three-dimensional search unit performs athree-dimensional search for specifying an attitude and a position ofeach workpiece from the input image acquired by the input imageacquisition unit by using the search model registered by the searchmodel registration unit.
 6. The robot setting apparatus according toclaim 4, wherein the search model registration unit and the workpiecemodel registration unit are configured by using a common member.
 7. Therobot setting apparatus according to claim 1, wherein the grip referencepoint setting unit sets the grip reference point to a presetpredetermined value and/or the grip direction setting unit sets the gripdirection to a preset predetermined value.
 8. The robot settingapparatus according to claim 1, wherein the grip reference point settingunit allows a user to set the grip reference point and/or the gripdirection setting unit allows a user to set the grip direction.
 9. Therobot setting apparatus according to claim 1, wherein a grip referencepoint and a grip direction passing through the grip reference point aredisplayed to overlap the end effector model in the image display region.10. The robot setting apparatus according to claim 1, furthercomprising: a workpiece grip propriety display region in which adetermination result of grip propriety at a grip position designated foreach workpiece in the three-dimensional pick determination unit isdisplayed; and a workpiece grip impossibility cause display region inwhich a cause of grip impossibility for a grip position which isdetermined as grip being impossible at the grip position designated foreach workpiece in the three-dimensional pick determination unit isdisplayed.
 11. The robot setting apparatus according to claim 10,wherein the three-dimensional pick determination unit includes aninterference determination unit that determines the presence or absenceof interference with a member present around a workpiece at a gripposition designated for the workpiece by the workpiece side griplocation designation unit on the basis of a search result of eachworkpiece searched for by the three-dimensional search unit, and whereinthe three-dimensional pick determination unit determines that theworkpiece determined as there being interference by the interferencedetermination unit cannot be gripped.
 12. The robot setting apparatusaccording to claim 11, further comprising: an inclined angle settingunit that sets an allowable inclined angle range for an attitude of aworkpiece, wherein the interference determination unit includes an angledetermination unit that determines whether or not an attitude of asearch result of a workpiece searched for by the three-dimensionalsearch unit is included in an inclined angle range set by the inclinedangle setting unit, and wherein the three-dimensional pick determinationunit determines that the workpiece cannot be gripped in a case where theangle determination unit determines that the attitude of the searchresult of the workpiece searched for by the three-dimensional searchunit is not included in the inclined angle range set by the inclinedangle setting unit.
 13. The robot setting apparatus according to claim9, wherein a cause of grip impossibility displayed in the workpiece gripimpossibility cause display region includes at least one of an endeffector model interfering with an object present around a workpiece andan inclined angle of an end effector model exceeding a predeterminedrange.
 14. The robot setting apparatus according to claim 1, wherein theworkpiece side grip location designation unit registers a plurality ofgrip positions for a workpiece model.
 15. The robot setting apparatusaccording to claim 1, further comprising: a grip solution candidatedisplay region in which grip positions set for any one of search resultsof one or more workpieces searched for by the three-dimensional searchunit are displayed in a list form.
 16. The robot setting apparatusaccording to claim 15, wherein a position and an attitude of an endeffector model corresponding to a grip position selected in the gripsolution candidate display region are displayed in the image displayregion.
 17. The robot setting apparatus according to claim 1, whereinthe workpiece side grip location designation unit displays, as aninitial value, a state in which the end effector model is disposed to bedirected downward, and the workpiece model is disposed under the endeffector model, in the workpiece display region, and, in this state,designates the grip position at which the end effector model grips theworkpiece model.
 18. The robot setting apparatus according to claim 1,wherein the workpiece model registered by the workpiece modelregistration unit is one of six fundamental direction images in whichthe workpiece model is viewed from positive and negative directions ofeach of a first axis, a second axis, and a third axis defining a virtualthree-dimensional space and orthogonal to each other.
 19. A robotsetting method of setting an operation of a robot performing a binpicking operation of a sensor unit measuring a three-dimensional shapeof each of a plurality of workpieces stacked in a work space andsequentially taking out the workpieces with an end effector provided ata tip of an arm portion of the robot, the robot setting methodcomprising: a step of displaying a workpiece model virtually expressinga three-dimensional shape of a workpiece with three-dimensional CAD dataor a height image, and an end effector model virtually expressing athree-dimensional shape of an end effector with three-dimensional CADdata, in an image display region representing a virtualthree-dimensional space; a step of designating a grip position at whichthe end effector model grips the workpiece model for the workpiece modeldisplayed in the image display region in a state in which a gripreference point corresponding to a position at which the workpiece modelis gripped and a grip direction in which the end effector model gripsthe workpiece model are defined for the end effector model; and a stepof automatically adjusting a relative position between the end effectormodel and the workpiece model such that the grip direction is orthogonalto a workpiece plane representing an attitude of the workpiece modeldisplayed in the image display region, and the grip reference point andthe grip position are located along the grip direction.